This is the accepted version of the journal article: Fang, Zhijie; López Peña, Antonio M. «Intention recognition of pedestrians and cyclists by 2D pose estimation». IEEE Transactions on Intelligent Trans- portation Systems, Vol. 21 Núm. 11 (2020), p. 4773-4783. 11 pàg. DOI 10.1109/TITS.2019.2946642 This version is available at https://ddd.uab.cat/record/274820 under the terms of the license 1 Intention Recognition of Pedestrians and Cyclists by 2D Pose Estimation Zhijie Fang, Antonio M. López. Abstract—Anticipating the intentions of vulnerable road users (VRUs) such as pedestrians and cyclists is critical for performing safe and comfortable driving maneuvers. This is the case for human driving and, thus, should be taken into account by systems providing any level of driving assistance, from advanced driver assistant systems (ADAS) to fully autonomous vehicles (AVs). In this paper, we show how the latest advances on monocular vision-based human pose estimation, i.e. those relying on deep Fig. 1: Our focus: are the pedestrians going to cross? Convolutional Neural Networks (CNNs), enable to recognize the intentions of such VRUs. In the case of cyclists, we assume that they follow traffic rules to indicate future maneuvers with arm signals. In the case of pedestrians, no indications can be assumed. Instead, we hypothesize that the walking pattern of a pedestrian allows to determine if he/she has the intention of crossing the road in the path of the ego-vehicle, so that the ego-vehicle must maneuver accordingly (e.g. slowing down or stopping). In this paper, we show how the same methodology can be used for recognizing pedestrians and cyclists’ intentions. For pedestrians, we perform experiments on the JAAD dataset. For cyclists, we Fig. 2: Cyclist arm signals, left to right: Turn Left, Turn Right, did not found an analogous dataset, thus, we created our own Alternative Turn Right and Stop. one by acquiring and annotating videos which we share with the research community. Overall, the proposed pipeline provides new state-of-the-art results on the intention recognition of VRUs. we would say that they are not in the path of the vehicle at the Index Terms—Autonomous Driving, ADAS, Computer Vision, moment. However, we would like to know what is going to Pedestrians Intention Recognition, Cyclists Intention Recognition happen next: is the stopped pedestrian suddenly going to cross the road? is the walking pedestrian going to cross the road without stopping?; in the affirmative cases, the vehicle could I. INTRODUCTION start to slow down already for a safer maneuver, increasing the comfort of the passengers and the confidence of the pedestrians E VEN there is still room to improve the detection and tracking of vulnerable road users (VRUs), i.e. pedestrians and cyclists, the state-of-the-art is sufficiently mature [1], [2], (especially relevant for autonomous vehicles). Recognizing the motion intentions of cyclists is also highly relevant since many times the ego-vehicle will need to overtake [3], [4], [5], [6], [7], [8] as to allow for increasingly focus them. While we cannot assume that pedestrians will explicitly more on related higher level tasks, which are crucial in terms indicate their intentions, for the cyclists we can exploit traffic of assisted and automated driving safety and comfort. rules. In particular, cyclists must indicate future left/right turns In particular, knowing the intention of a pedestrian to cross and stop maneuvers with arm signals (see Fig. 2). the road in front of the ego-vehicle, i.e. before the pedestrian has actually entered the road, would allow the vehicle to warn In this paper, we explore the idea of using 2D pose the driver or automatically perform smoother maneuvers more estimation from monocular images as core information to respectful with pedestrians; it even significantly reduces the recognize the intentions of both pedestrians and cyclists. In chance of injury requiring hospitalization when a vehicle-to- fact, we already addressed the pedestrian crossing/not-crossing pedestrian crash is not fully avoidable [9]. The idea can be classification (C/NC) task by relying on image-based 2D pose illustrated with the support of Fig. 1. We can see two pedestri- estimation [10], [11]. The proposed method shows state-of-the- ans, one apparently stopped near a curb and the other walking art results and, in contrast to previous approaches (see Sect. towards the same curb. Just looking at the location of the II), it does not require information such as stereo, optical flow, (yellow) bounding boxes (BBs) that frame these pedestrians, or ego-motion compensation. In the most recent work [11], we reported results for the (Joint Attention for Autonomous Zhijie and Antonio are with the Dpt. of Computer Science at the Universitat Driving–JAAD) dataset [12], which allows to address the Autònoma de Barcelona (UAB), and with the Computer Vision Center (CVC) C/NC classification task in naturalistic driving conditions. at the UAB. e-mail: {zfang, antonio}@cvc.uab.es We acknowledge the financial support by the Spanish project TIN2017- Moreover, since recently CNN-based features have been used 88709-R (MINECO/AEI/FEDER, UE). Antonio thanks the financial support to address the C/NC classification task in JAAD [13], we by ICREA under the ICREA Academia Program. Zhijie thanks the Chinese additionally compared our pose-estimation-based features with Scholarship Council, Grant 201406150062. We thank the Generalitat de Catalunya CERCA Program and its ACCIO agency. We thank the student CNN-based ones, the former clearly outperforming the latter. Arjun Gupta who annotated part of CASR during an internship at CVC. Overall, [11] reported the new state-of-the-art baseline for 2 JAAD. In this paper, first we complement this work by relied on a monocular stream of frames, neither on stereo, nor analysing the effect of obtaining noisy 2D skeletons, and we on optical flow, nor on ego-motion compensation. In this paper, report also the features that the C/NC classifier considers most we augment our study to the more challenging JAAD dataset relevant. In addition, we show how the same methodology can by complementing the 2D pose estimation with state-of-the- be used to recognize cyclist arm signals; which also required to art pedestrian detection and tracking. Moreover, we compare collect and annotate a specific dataset presented in this paper the use of skeleton-based features with CNN-appearance-based too. As in JAAD, our cyclist arm sign recognition (CASR) ones as suggested in [23] for the generic task of human action dataset was collected with a consumer-graded camera. We recognition. We will see how the former bring more accuracy have annotated 219 arm signal actions on videos of approx- than the latter. We also report time-to-event results. imately 10 seconds each, containing one or two actions per Pedestrian intention recognition is reviewed in [24] for video. We also annotated 10 additional Youtube videos. CASR technologies such as LiDAR and wifi. Compared to pedestrian is publicly available in https://github.com/VRU-intention/casr. intention recognition, recognizing cyclist arm signals has re- Sect. II reviews the related work. Sect. III presents our ceived less attention. One reason may be the lack of publicly proposal to recognize VRU intentions. Sect. IV and Sect. V available datasets to assess this task. After [3], [5], it was detail the experiments, results, and derived conclusions. We publicly released a large dataset focusing on cyclists, termed present CASR in Sect. V. Finally, Sect. VI summarizes the as Tsinghua-Daimler Cyclist Benchmark dataset (TDCB); work and its possible continuations. however, acquired data and annotations are intended to support detection and orientation estimation tasks, but not cyclist arm signal recognition. In [6], the ground truth of TDCB was II. RELATED WORK extended with wheel annotation for the case of bikes in side C/NC classification started as a pedestrian path predic- view, still to support cyclist detection. Thus, in this paper, we tion problem; addressed by relying on pedestrian dynamic introduce our Cyclist Arm Signal Recognition dataset (CASR) models for estimating pedestrian future location, speed and containing 40,218 frames organized as short videos containing acceleration [14], [15]. However, these models are difficult to a cyclist arm signal each, in total, 219 annotated actions. adjust and for robustness require to rely on dense stereo data, For assessing generalization we also annotated 10 additional dense optical flow and ego-motion compensation. Intuitively, actions from Youtube, corresponding to 1,636 frames. methods like [15] implicitly try to predict how the silhouette Using a stereo rig, in [8], [25] it is detected whether the of a tracked pedestrian evolves over time. In fact, [16] uses a left arm of a cyclist observed from the back is up or down, stereo-vision system and ego-motion compensation to explic- which is used as a context cue within a path prediction module. itly assess the silhouette of the pedestrians (others rely on 360◦ However, an isolated accuracy analysis of such up/down arm LIDAR [17]). Note that, while our method will be applied in classification is not performed. In order to perform such a JAAD because only relies on a monocular stream of images, classification, the disparity map computed from stereo image these other methods cannot be applied due to the lack of stereo pairs is used to produce a binary mask of each detected cyclist, information and vehicle data for ego-motion compensation. and template matching is applied to determine if the mask On-board head and body orientation approximations have correlates with a left arm up or down. In particular, the scores been also proposed to estimate pedestrian intentions, both from of matching against multiple templates, the disparity values, monocular [18] and stereo [19], [20] images with ego-motion and the image intensities, are used as core information to compensation. However, it is unclear how we actually can use build a Naive Bayesian Classifier with uniform prior, which is these orientations to provide intention estimation. Moreover, responsible for the desired up/down arm classification. In this the experiments reported in [20] suggest that head detection paper, we do not assume stereo data and we not only account is not useful for the C/NC classification task. for the cyclist signal to turn left, but also to turn right (two These mentioned vision-based works relied on Daimler’s types as shown in Fig. 2) and stopping. Moreover, we apply dataset. By using an AlexNet-based CNN trained on JAAD, exactly the same procedure for pedestrian intention recognition [13] verified whether full body appearance improves the results and for cyclist arm signal recognition. On the other hand, the on the C/NC classification task compared to analyzing only classification output of our method could be also integrated as the sub-window containing either the head or the lower body. a cue for the path prediction module of [8], [25]. Conclusions were similar, i.e. specifically focusing on legs or Other concurrent works to ours, have been recently pre- head does not seem to bring better performance. sented also exploring the use of skeleton-based features. In In fact, a lack of information about the pedestrian’s posture [26], Gaussian processes operating on fitted human skeletons and body movement results in a delayed detection of the are used to recognize four types of actions, namely walking, pedestrians changing their crossing intention [21]. In line stopping, starting, and standing. Results on naturalistic pub- with this suggestion, in [10] we relied on a state-of-the- licly available driving datasets (i.e. such as JAAD) are not art 2D pose estimation method that operates in still images reported and cyclist arm sign recognition is not addressed. In [22]. In particular, following a sliding time-window approach, [27], fitted skeletons are analyzed along time using a long accumulating estimated pedestrian skeletons over-time (see short-term memory (LSTM) for detecting actions such as Fig. 3) and features on top of these skeletons (see Fig. 8), crossing, stopping, starting, turning, and walking along. In this we obtained state-of-the-art results for the C/NC classification case, sequences in the wild are used but, up to the best of our task in Daimler’s dataset; which is remarkable since we only knowledge, they are not publicly available so it is difficult 3 Fig. 3: 2D skeleton fitting on 14 consecutive frames of one JAAD sequence, which roughly correspond to half a second. to compare them with JAAD for instance. On the other hand, and has been trained on the Microsoft COCO 2016 keypoints cyclist arm sign recognition is not addressed. As an aside note, dataset [34]. Thus, a priori it could be effective to fit the pose even we have not included in this paper, we also experimented of both pedestrians and cyclists. In fact, this method is sup- with LSTMs but the obtained results were not better than the posed to perform both human detection and pose estimation. ones that we will report in this paper. In [28], the main focus However, as we will comment in Sect. IV, for our problem it is to propose a human pose extraction method, the further was more effective to rely on a detector-tracker pipeline and analysis of how to use it for automatic action recognition then run the pose estimation module only inside the tracked is not considered, only a manually guided visual analysis is BBs, obtaining in that way the desired skeletons (see Fig. 3). performed and, therefore, no quantitative results are reported. d) Intention classification: Focusing on the C/NC clas- Finally, we would like to mention US Patent [29]. In the sification task, in [10] we extracted features from fitted pedes- described approach, arm signal recognition is based on LiDAR trian skeletons and use them as input to a shallow classifier. data, while we rely only on monocular vision. [29] does not Fig. 4 highlights with stars the 9 keypoints we found as report results on any specific dataset; however, we think this most stable, which correspond to the legs and the shoulders. LiDAR-based approach and ours can be complementary. These are highly relevant keypoints since the legs execute continue/start walking or stopping actions; while keypoints III. METHOD from shoulders and legs inform about global body orientation. We need to detect pedestrians, track them, adjust a skeleton From the selected keypoints we compute features. First, we for each one (Fig. 3), and apply a C/NC classifier that perform a normalization of keypoint coordinates according to relies on skeleton-based features (Fig. 8). Cyclist arm signal a factor h proportional to the pedestrian height, determined classification follows analogous steps. as the vertical distance from the top-most to the bottom- a) Detection: we can use a state-of-the-art pedes- most keypoints (their location depends on the skeleton fitting). trian/cyclists (object) detector as long as it only requires a Then, different features (conveying redundant information) are single RGB image as input, and returns a set of bounding computed by considering distances and relative angles between boxes (BBs), each one framing a pedestrian/cyclist (by cyclist, pairs of keypoints, as well as triangle angles induced by triplets we mean that the BB can frame only the rider or both the rider of keypoints (see Fig. 8). In total we obtain 396 features. and the bike). Due to their popularity, here we have considered Since we concatenate features during the last T frames, our Faster R-CNN [2] and Mask R-CNN [30], which can be found feature vector has dimension 396T . Finally, we use a Random in both the Detectron [31] and TensorPack [32] frameworks. Forest (RF) classifier, which directly provides a probability for b) Tracking: we selected [7], an efficient state-of-the-art a meaningful thresholding to perform the C/NC classification. multiple object tracker, with publicly available code. It uses On the other hand, the pose of a cyclist is rather different the following state for a detected object: (u, v, λ, h, ẋ, ẏ, λ̇, ḣ); than the pose of a pedestrian walking or standing. However, where (u, v) is the central pixel of the BB, λ its aspect ratio, we hypothesize that, for performing arm signal classification, h its height, while ẋ, ẏ, λ̇, and ḣ are the respective derivatives we can rely on the same keypoints than for C/NC classification over time. These variables are updated by Kalman filtering. plus the two additional keypoints from each arm (elbow and For data association, it is used a cosine distance on top of wrist, see Fig. 4). Therefore, for each cyclist we use 13 CNN features (trained on a large-scale person re-identification keypoints, which turns out in 1170 features per frame, thus, dataset [33], thus, especially useful for tracking pedestrians 1170T for a temporal (sliding) window of T frames. and cyclists) which scores the degree of visual similarity between BB detections and predictions. A detection which IV. PEDESTRIAN INTENTION EXPERIMENTS does not have a high matching score with some prediction is pre-tracked; if the lack of matching holds during several A. Dataset consecutive frames, the track is consolidated as corresponding First publicly available dataset for research on detecting to a new detected object. Predictions which do not have a high pedestrian intentions is from Daimler [14]. It contains 68 short matching score with a new detection during several frames are sequences (9,135 frames in total) acquired in non naturalistic considered as disappeared objects (ended tracks). conditions and shows a single pedestrian per video, where the c) Skeleton fitting (pose estimation): Given the good pedestrian performs pre-determined actions. More recently, it results obtained in [10], we apply the CNN-based pose esti- has been publicly released the Joint Attention for Autonomous mation method proposed in [22], which has publicly available Driving (JAAD) dataset [12], acquired in naturalistic condi- code. This method can operate in still monocular images tions and annotated for detecting C/NC actions. It contains 4 Fig. 4: Keypoints used for detecting the intentions of pedestri- ans (left) and cyclists (right). In the former case, 9 keypoints Fig. 5: C/NC classification. The ground truth label is indicated that are used out of the fitted skeleton to extract 396 features. with ”C” or ”NC”; In green if the prediction agrees with the In the later case, 13 keypoints are used to extract 1170 features. ground truth, red otherwise. Pedestrians are framed with two BBs: detection and tracking, the latter with the track ID. The estimated pedestrian skeleton is also shown. When annotated, 346 videos (most of them 5-10 seconds long) recorded on- time-to-event (TTE) is also shown in frame units. Negative board with a monocular system, running at 30 fps with a TTE values mean that the event happened before this frame, resolution of 1920 × 1080 pixels. Videos include both North while positive values indicate that it will happen after. America and Eastern Europe scenes. Overall, JAAD includes ≈ 88,000 frames with 2,624 unique pedestrians labeled with ≈ 390,000 BBs. Moreover, occlusion tags are provided for Moreover, for a tracked pedestrian, these conditions must hold each BB. Where ≈ 72,000 (18%) BBs are tagged as partially over more than T frames, since we need to concatenate last T occluded and ≈ 46,000 (11%) as heavily occluded. Pedestrian frames for the C/NC classification. Thus, from tracks longer tracks are also provided. In addition, although we are not using than T frames we can obtain different training samples by it in this paper, JAAD contains also context information (traffic applying a temporal sliding window of size T. signs, street width, etc.) that we may use in further studies to complement purely pedestrian-based information. For each tracked pedestrian, the C/NC label assigned to a sequence of length T corresponds to the label in the most recent frame (i.e. the frame in which the C/NC decision must B. Evaluation protocol be taken). We set T=14 for JAAD (i.e., following [10], a value In [13], JAAD was used for evaluating a C/NC classifier. roughly below 0.5 seconds). Since we are in training time, However, neither is explained how JAAD was split in training here we are referring to the ground truth tracks provided in and testing, nor the code is available. Here we use the first JAAD. For completeness, we also test the case T=1; meaning 250 videos of JAAD for training and the rest for testing. We that we only train with the last frame of the same sequences have re-labeled as C the crossing labels of JAAD, as well used for the T=14 case. Overall, there are 8,677 sequences as the labels in {clear-path, moving-fast, moving-slow, slow- of length T=14 and NC label, while there are 36,253 with C down, speed-up} assigned to a pedestrian with lateral motion label; thus, in the latter case we randomly take only 8,677 direction; the rest JAAD labels are re-labeled as NC. among those 36,253 possible. Accordingly, we fit the pose 1) Training: While pedestrian detection based on the skele- estimation-based skeleton and compute the C/NC features for ton fitting algorithm that we use here [22] is possible, in our 8,677 C and 8,677 NC samples (in a set of experiments for initial experiments we determined that its detection accuracy in T=14, in another for T=1). These features are then used as JAAD was worse than using Faster R-CNN since the latter was input for the scikit-learn [36] function GridSearchCV; which fine-tuned from the available pedestrian BB annotations (fine- is parameterized for training a Random Forest (RF) classifier tuning the CNN of [22] would require body level annotations, using 5-fold cross-validation with the number of trees running not available in JAAD). In particular, we used the Faster R- on {100, 200, 300, 400, 500} and maximum depth running CNN implementation released in [35], following the same on {7, 15, 21, 30}. The optimum RF in terms of accuracy Faster R-CNN settings but using {8, 16, 32, 64} as anchors corresponds to 400 trees and a maximum depth of 15, but and 2.5 as BB aspect ratio (i.e. pedestrian oriented). This we noted that all configurations provided similar accuracy. Faster R-CNN is based on VGG16 CNN architecture. For fine- Since we use Faster R-CNN, we can compare skeleton- tuning we perform 110,000 iterations (remind that an iteration based features with CNN-based ones, as is common in general consists of a batch of 256 regions from the same image, and action recognition literature [23]. Accordingly, we apply the that input images are vertically mirrored to double the number following procedure. For all training images we run the of training samples). Regarding learning rate, we start with VGG16 obtained during Faster R-CNN fine-tuning. Then, for 0.001 and decrease the value to 0.0001 after 80,000 iterations. the same tracks mentioned before, we replace the skeleton- For training the C/NC classifier we rely on well-seen pedes- based features by the fc6 layer features inside the tracked trians and we balance the number of samples of the C and NC pedestrian BBs. Note that (Sect. III) we have 396T skeleton- classes. Thus, we only consider pedestrian training samples based features and 4096T fc6-based ones for each sample with a minimum BB width of 60 pixels and no occlusion. reaching RF training. In terms of RF parameter optimization 5 (number of trees and maximum depth), CNN-based features TABLE I: Classification accuracy (Acc) in JAAD. SKLT refers reported similar accuracy as was the case for skeleton-based to our skeleton-based features, while CNN (fc6) are features ones. Therefore, we set the same parameters, i.e. 400 trees from a VGG16 fine-tuned in JAAD (see main text). We have and a maximum depth of 15. We also combine skeleton and included here the results reported in [13], where CNN features CNN-based features using the same RF parameters. are based on a non-fine-tuned AlexNet and Context refer 2) Testing: In [13], evaluations are single frame (T=1) and to features of the environment, not of the pedestrian itself. only pedestrians with an action label are considered (mapped Moreover, results for 20% and 30% noise in the keypoints is to C/NC here). When designing our experiments, we have seen also reported for the SKLT case (se main text for details). that not all pedestrians of JAAD are annotated with a BB. Thus, when we run the detection and tracking modules, we M ethod T features Acc Acc Acc 20% 30% are detecting and tracking some pedestrians which do not have the required ground truth information (BB, etc.). So, in order [13] 1 CN N 0.39 [13] 1 CN N &Context 0.63 to follow a similar approach to [13], we do not consider these cases for quantitative evaluation. However, they are present Ours 1 CN N (f c6) 0.68 Ours 1 SKLT 0.80 0.77 0.73 in the qualitative evaluation (e.g. see the videos provided as Ours 1 CN N (f c6) + SKLT 0.81 supplementary material). Overall, we ensure that T=1 and Ours 14 CN N (f c6) 0.70 T=14 experiments are applied at the same tracked pedestrians Ours 14 SKLT 0.88 0.86 0.83 at the same frames, performing a fair comparison. Ours 14 CN N (f c6) + SKLT 0.87 When detecting pedestrians with Faster R-CNN we use the default threshold 5% and overlapping of 30% for non- maximum suppression. For starting a new track, a pedestrian C. Results must be detected in 3 consecutive frames (i.e. 0.1 seconds); while for ending a track there must be no new matched Table I reports accuracy results. We have included those re- observations (detections) during 30 frames (i.e. 1 second). For ported in [13]; however, our results are not directly comparable pose estimation (skeleton fitting) we use 3 scales; in particular, since it is unclear which frames were used for training and {1, (1 − 0.15), (1 − 0.15 ∗ 2)}. For the C/NC classifier, we which ones for testing. The paper mentions that heavily oc- threshold in 0.5 the probability value provided by the RF. cluded pedestrians are not considered for testing. In our exper- We assess accuracy according to the widespread definition imental testing we do not exclude pedestrians due to occlusion. Acc = (T P + T N )/(P + N ), where P stands for total pos- Moreover, we also report TTE information. However, we still itives (here ”C”), N are the total negatives (here ”NC”), and found interesting to include the results in [13] since the paper T P and T N the rightly classified positives and negatives (C is based on CNN features and T=1. In particular, the authors and NC right classifications). According to the testing protocol train a walking/standing classifier and another looking/not- we have defined, we found P = 17045 and N = 5161, looking (pedestrian-to-car) classifier, both classifiers are based therefore, Acc could be bias towards ”C” results. In order to on a modified AlexNet CNN. The classification score of these avoid this, we select P = N cases randomly. Thus, Acc will classifiers is not used for final C/NC decision. Instead, the be based on 10,322 testing decisions. fc8 layer of both are used as features to perform a final As in [10], we are interested in time-to-event (TTE) results C/NC based on a Linear-SVM adjusted in such a CNN-based for the critical case of crossing (C). However, JAAD is not feature space. It is also proposed to add contextual information annotated for this. Thus, we added TTE information to 9 keep- captured by a place-classification style AlexNet. walking-to-cross sequences, and to 14 start-walking-to-cross For a fixed T, Table I shows that the skeleton-based features ones. TTE = 0 is the when the event of interest happens. For (SKLT) outperform those based on CNN fc6 layer. Combining keep-walking-to-cross, it is the first frame at which the trunk of SKLT and fc6 does not significantly improves accuracy of the walking pedestrian is over the curbside. For start-walking- SKLT. We can see also that T=14 outperforms T=1, showing to-cross, it is the frame at which the stopped pedestrian starts the convenience of integrating different frames. Fig. 6 shows moving a leg forward. Positive TTE values correspond to how the system is stable at predicting that a walking pedestri- frames before the event, negative values to frames after the ans will keep moving from a sidewalk and eventually crossing event. Fig. 5 shows a result example where we can see TTE the curbside appearing in front of the vehicle. We can see values for different pedestrians that are correctly classified also that we can predict (predictability>0.8) that a standing as crossing (the supplementary videos have more examples). pedestrian will cross the curbside around 8 frames after he/she With TTE we provide two different plots, intention probability starts to move, which in JAAD is around 250ms. vs TTE, and predictability vs TTE. With the former we can Looking in more detail to the results, we find situations that see how many frames we can anticipate the pedestrian action. need to be taken into account as future work. For instance, in Since there are several testing sequences per intention, mean Fig. 7 there is a ”C” accounted as error (red). Indeed, the and standard deviation are plotted. Predictability plots show a pedestrian is crossing the road, but not the one intersecting normalized measurement of how feasible is to detect the action the path of the ego-vehicle. So in the evaluation it should be under consideration for each TTE value. Predictability zero probably accounted as right. On the contrary, in Fig. 7 the indicates that we cannot detect the action, while predictability system classifies as ”NC” a pedestrian which is not crossing one means that we can. the road, but in fact is walking along the road, in front of 6 TABLE II: For T=1, top-25 most relevant pedestrian skeleton- TABLE IV: Cyclist arm signals in CASR and some YT videos. based features from left-to-right and top-to-bottom. Turn Left Turn Right Stop Θ(4, 12, 3) Θ(4, 11, 3) Θ(8, 4, 9) Θ(4, 8, 3) Θ(3, 10, 5) Cyclist 1 24 38 34 Θ(8, 12, 10) Θ(12, 11, 13) Θ(3, 12, 5) Θ(4, 13, 3) Θ(3, 8, 5) Cyclist 2 16 24 30 Θ(4, 10, 3) Θ(8, 5, 9) Θ(8, 3, 9) Θ(3, 11, 5) Θ(12, 10, 13) Cyclist 3 6 12 26 Θ(4, 9, 3) Θ(10, 8, 12) Θ(3, 9, 5) Θ(8, 10, 12) Θ(3, 8, 9) Cyclist 4 2 3 4 Θ(3, 9, 8) Θ(3, 13, 5) Θ(9, 13, 11) Θ(12, 9, 13) Θ(12, 8, 13) Total CASR 48 77 94 Total YouTube 6 4 0 TABLE III: Analogous to Table II for T=14. 7 Θ (4, 9, 3) Θ3 (4, 12, 3) Θ8 (4, 13, 3) Θ9 (4, 12, 3) Θ2 (4, 12, 3) CASR we followed a similar approach than JAAD authors. In Θ6 (4, 9, 3) Θ12 (4, 8, 3) Θ9 (4, 10, 3) Θ4 (4, 10, 3) Θ1 (4, 9, 3) Θ4 (4, 8, 3) Θ8 (4, 12, 3) Θ6 (4, 8, 3) Θ8 (4, 10, 3) Θ6 (4, 13, 3) particular, we attached a GoPro camera to the windshield of a Θ6 (8, 10, 9) Θ1 (4, 12, 3) Θ4 (3, 8, 5) Θ2 (4, 8, 3) Θ3 (4, 8, 3) car, forward facing the road ahead. We set the acquisition to Θ7 (4, 8, 3) Θ4 (4, 9, 3) Θ7 (4, 10, 3) Θ5 (4, 11, 3) Θ8 (4, 9, 3) RGB images at 30 fps and an 1920 × 1080 resolution. We asked four persons to drive their bikes inside our university campus, and they were instructed to ride around the car. Now this situation is accounted as right, but probably as they wish but using arm signals when required. Sometimes should be accounted as wrong. On the other hand, in this they wear helmet, sometimes not. Sometimes they carry a bag case we can just use location-based reasoning to know that in their back, sometimes not. YouTube videos are also based on the pedestrian is in a dangerous place, it is not a problem a dash cam facing the road. Table IV summarizes the number of predicting the action anymore (as the C/NC case). It is of actions (cyclist arm signals) that we have annotated. Note worth also mentioning that we have observed that walking that CASR includes 219 annotated actions, and YouTube 10. in parallel to the car motion direction, tends to be properly Actions have been organized as short videos of around 10 classified as NC; however, more annotations are required to seconds with a single cyclist, where the frame starting an provide a reasonable quantitative analysis. action and the frame ending this action are annotated. The In order to evaluate the robustness of the method, we ran videos of CASR mostly show one action and sometimes two an equivalent set of experiments for the SKLT case, where we actions because they were indicated in a continuous way by the added random noise to the keypoints of the fitted skeleton in cyclists, and in this case we did not split the video. In addition testing time. In particular,independently for each coordinate to the frame level action annotations, we have annotated the of each keypoint, we added Gaussian noise with zero mean 2D BBs framing the cyclists too. Moreover, the videos where and standard deviation s which, following [37], is set as a selected so that in most of them no pedestrians are included; percentage over the distance to the closest keypoint. This thus, ready to focus on cyclist arm signal recognition. In some is shown in Table I for percentages of 20% and 30%. As cases, however, there can be some pedestrians but we do not expected, accuracy decreases a few for 20% and more for annotate his/her BB so that they are ignored during training 30%, being T=14 is more robust to noise than T=1. and testing. Overall, CASR’s content is analogous to the first Finally, we assess the most important features for the RF deployed dataset for pedestrian intention recognition [14], but classifier. In Fig. 8 we assign an ID to the keypoints of a including much more annotated frames (68 actions within fitted skeleton either used for pedestrian or cyclist intention 9,135 in [14], 219 actions within 40,218 frames here). recognition. The visualized naming scheme defines angles Note how action annotations are vehicle-centric here, in- (Θ(·, ·), Θ(·, ·, ·)) and lengths (L(·, ·), Lx (·, ·), Ly (·, ·)). Since stead of cyclist-centric. When the ego-vehicle follows the we have evaluated both T=1 and T=14, in the latter case we cyclist, they are the same. However, when the cyclist and the also add a super-index to indicate from which relative frame ego-vehicle move in opposite directions, we annotated as left- index (i.e. in {1, .., 14}) comes the feature. Tables II-III show turn what for the cyclist is an indication of right-turn, and vice the top-25 more important features for T=1 (top ∼ 6%) and versa. The reason, is that for the vehicle what matters is the T=14 (top ∼ 0.5%), respectively. We see how all are based direction that the cyclist is going to take as seen in the image on 3-keypoint angles, mostly connecting either shoulder and to be processed. Figure 9 clarifies the idea. legs, or shoulder and waist; thus, capturing global pose. For T=14, only one feature appears after frame 9 (∼ 300ms); thus, favoring intention prediction in a short time. B. Evaluation protocol In CASR we recorded four cyclists. Accordingly, in order V. CYCLIST INTENTION EXPERIMENTS to evaluate our arm signal classifier, we divide their videos A. Dataset in training, validation, and testing sets. We use the videos There are large datasets for cyclist detection such as the of two cyclists for training, the videos of the other two already mentioned TDCB [3], [8]. However, it does not include cyclists are used for validation (training time) and testing, samples with annotations to assess arm signal recognition. respectively. By varying the role of the cyclists, we can Therefore, in this paper, we introduce our Cyclist Arm Signal perform 12 training-validation-testing runs. Moreover, for each Recognition dataset (CASR), consisting of 40,218 frames. trained classifier, we test on the annotated YouTube videos Moreover, for assessing generalization we also annotated ad- too. We report individual metrics for each trained classifier, ditional videos from YouTube, consisting of 1,626 frames. For as well as averaged metrics. Since we aim at performing 7 Fig. 6: For T=14, left to right: (a) Action prob. for Keep walking to cross, mean over sequences and standard deviation; (b) Same for Start crossing; (c) Predictability with action prob. thr. = 0.5, for Keep walking to cross; (d) Same for Start crossing. (a) Annotated as turning right Fig. 7: Results of C/NC classification (b) Annotated as turning left (c) Annotated as stopping Fig. 9: Annotation of cyclist arm signals. We have followed a Fig. 8: Skeleton-based features: angles and distances. vehicle-centric criterion for left/right annotation. TABLE V: Classification accuracy (Acc) and F1 score, both ranging from 0 to 1. Worst, best and average (with standard TABLE VI: Average classification accuracy (Acc) and F1 deviation) results over 12 experimental runs for each T value. score, for both CASR and YouTube videos (-YT). Results are We report generalization results on YouTube videos (-YT). reported for noise free keypoints, i.e. using them as provided by the skeleton fitting algorithm, as well as for two different T=1 Acc F1 Acc-YT F1-YT levels of noise (20% and 30%) on their location, which is Worst 0.89 0.86 0.82 0.78 forced at testing time (main text for details). Best 0.96 0.96 0.83 0.77 Avg 0.93 ± 0.02 0.92 ± 0.03 0.82 ± 0.01 0.76 ± 0.02 T=1 Acc F1 Acc-YT F1-YT T=14 Acc F1 Acc-YT F1-YT N oisef ree 0.93 ± 0.02 0.92 ± 0.03 0.82 ± 0.01 0.76 ± 0.02 20% 0.91 ± 0.02 0.90 ± 0.03 0.81 ± 0.01 0.75 ± 0.02 Worst 0.89 0.87 0.83 0.79 30% 0.87 ± 0.02 0.86 ± 0.03 0.80 ± 0.01 0.72 ± 0.02 Best 0.96 0.96 0.85 0.79 Avg 0.93 ± 0.02 0.92 ± 0.03 0.83 ± 0.02 0.77 ± 0.02 T=14 Acc F1 Acc-YT F1-YT N oisef ree 0.93 ± 0.02 0.92 ± 0.03 0.83 ± 0.02 0.77 ± 0.02 20% 0.93 ± 0.02 0.92 ± 0.02 0.83 ± 0.02 0.77 ± 0.02 30% 0.92 ± 0.02 0.92 ± 0.02 0.83 ± 0.02 0.76 ± 0.02 per-frame arm signal classification, we use the F1 and Ac- curacy standard metrics by counting classification errors and successes in each tested frame. For each training-validation run, we performed RF hyper-parameter search for the number sufficient to perform our evaluation, i.e. we do not need to run of trees and the maximum depth allowed before performing a tracker. Moreover, since the human pose estimation method the corresponding testing. For the former, we validated over that we use [22] performs the double task of searching the the set {50, 100, 200, 300, 400, 500}, and for the later over the human and fitting its skeleton in a given 2D BB, we first relied set {7, 10, 13, 16, 19, 22, 25, 28, 31, 34}. In this case, we also on the fine-tuned Faster R-CNN detector described in Sect. IV consider T=1 and T=14 as temporal sliding window sizes. for providing such BBs. However, it tends to leave the arms Since CASR and the annotated YouTube videos contain of the cyclists out of the BBs. Thus, since our focus is not a single cyclist and no pedestrians, detecting the cyclist is on detection, we changed to Mask R-CNN [30], in this case 8 we did not fine-tuned the detector to CASR since object-level TABLE VII: For T=1, top-25 most relevant cyclist skeleton- silhouette ground truth would be required. In practice, Mask based features from left-to-right and top-to-bottom. R-CNN is providing accurate detections in CASR, and only Θ(4, 6, 7) Θ(9, 6, 7) L(8, 7) L(10, 7) Lx (10, 1) in a few frames the corresponding detections were missing. In Θ(3, 6, 7) Lx (10, 11) L(10, 6) Θ(3, 2, 7) Θ(6, 3, 7) L(3, 2) Θ(11, 5, 6) Θ(4, 7, 6) Θ(10, 6, 7) Θ(5, 6, 13) these frames, since we are not running a tracker, we just took a Θ(5, 6, 12) Θ(8, 6, 7) Θ(11, 6, 7) Θ(3, 7, 6) Θ(4, 2, 7) noisy version of the ground truth BB as detection. In particular, Θ(6, 2, 7) Θ(12, 6, 7) Θ(5, 6, 10) L(5, 9) Θ(3, 6, 12) we added uniform noise to the BB’s corner coordinates, being the amount of noise proportional to the size of the BB (we TABLE VIII: Analogous to Table VII for T=14. added independent Gaussian noise to each corner coordinates, L14 (8, 7) L12 x (10, 1) L8 (8, 7) L9x (10, 1) Θ14 (12, 5, 6) with zero mean and st. dev. set to 10% of the BB height, L5x (10, 1) L10 L6x (10, 1) L13 (5, 9) L6 (8, 7) x (10, 1) width, x and y coordinates). With this protocol we focus on L13 (10, 6) L2x (10, 1) L13 x (10, 1) Θ14 (3, 6, 2) Θ12 (2, 3, 7) the cyclist arm signal recognition itself. Moreover, Sect. IV L8x (10, 1) 14 Θ (8, 6, 7) Θ12 (4, 2, 7) L7x (10, 1) L12 (5, 9) L8 (5, 9) Θ14 (11, 5, 6) L7 (4, 11) L9x (10, 11) L10 (8, 11) shows that the conclusions on pedestrian intention recognition that we draw in [10] using a dataset designed just for such a task, also hold in naturalistic traffic conditions. Thus, we expect the same for cyclist arm signal recognition. In this case, we are able to understand the stop indication because of the hand, which is not involved in the analysis of the image. Therefore, this may suggest that a via to explore in C. Results future work could be to analyse the area of the image in the Table V shows the quantitative results for T = 1 (single extreme of the fitted arms. In the last case, we cannot see any frame), and T = 14 (roughly half a second), respectively. action in this particular frame, while the system indicates a These confirm the effectiveness of the proposed method with right turn. In fact, in the previous frames, the cyclist actually relatively high accuracy values, which are quite stable (very indicates a right turn; thus, overall this error is more due to low standard deviation). Testing in the YouTube videos is the fact that annotating the starting and ending of a given more challenging, but still the accuracy is remarkable since action can have a couple of frames of ambiguity. Therefore, we trained the model on CASR cyclists. In both cases, in practice, not detecting any action in this frame or a right aggregating temporal information does not help significantly turn probably must be considered as correct. Fig. 15 shows to boost performance; which can be expected since it is already error cases in the YouTube videos for T=14. From left to right, possible to understand what the intentions of the cyclists are one case due to having the cyclist arm indicating the action looking at a single frame. Still, analyzing more frames can out of the BB, two cases due to a bad fitting of the skeleton help to stabilize the classification output as we are going to because adverse conditions (bag in the back, narrow BBs and see. In order to confirm this, we force errors in the keypoints low contrast arm-background), and two cases where the action as we did for pedestrian intention recognition (see Table I). has just started and it is not yet clear enough (e.g. the last case Table VI compares overall accuracies for noises of 20% and is just the starting of the left-turn action correctly classified in 30%, both for T=1 and T=14. As we can see, only 30% causes the left example of Fig. 13). We also checked the confusion an appreciable drop on performance for T=1, which is avoided matrices for CASR results, T=1 and T=14. We did not find up to a large extent thanks to the multiframe setting, i.e. T=14. any particular confusion pattern between classes, neither for Figures 10-13 present examples of correct results for T=14. noise free experiments, nor with 20%/30% noise. The blue BB is the detection. Note how the predictions works Finally, we assess the most important features for the RF for forward and backward faced pedestrians, even if they carry classifier. Table VII shows the case T=1, for its Best classifier a bag in the back, and at different distances (bigger characters in Table V. Most features correspond to angles defined by correspond to further away detections). Fig. 14 shows some either a keypoint from neck/shoulders/waist/legs and two kei- isolated frames with wrong predictions for CASR and T=14. points from arms (e.g. Θ(4, 6, 7)), or two keypoints from the From left to right, the two first cases correspond to oncoming former set and one from the later (e.g. Θ(3, 2, 7)). Distances cyclists indicating the intention of stopping and turning to between these two sets are also among the most relevant (e.g. their right (left in vehicle-centric coordinates), but no action L(8, 7)). Table VIII is for T=14 and its Best classifier of Table is recognized because the detection BBs left the arms out V. We see how current frame (i.e. frame 14 of the temporal affecting the fitting of the skeleton. This could be solved by sliding window) mainly contributes with angle-based futures, systematically augmenting the BB size which is taken as area which is coherent with the results of T=1; i.e. to favour early of interest by the skeleton fitting procedure, at testing time. recognition despite using more frames. We see also how there In the third case the system confuses a future turn left with a are many distance features between neck/shoulder/waist/leg stop indication, however, this is the case only at the starting and arm keipoints (e.g. L8 (8, 7), L9x (10, 1)), most of them are of the action because it is not really clear what the cyclist is concentrated in the middle of the temporal window (from going to indicate. The next frames make it clear so that the frame 5 to 10 we find 12 feature-based features from the system actually predicts the proper maneuver. In the fourth top-25, from frame 12 to 14 we find 6) which makes clas- case, the system recognises that the cyclist is indicating an sification more stable once the cyclist has indicated the sign action, however, a stop sign is confused with a turn left, which for ∼ 200 − 300ms. Note also how for cyclists, 25 features happens because of the relatively straight position of the arm. are the ∼ 2% for T=1, and ∼ 0.15% for T=14. 9 Fig. 10: Correct predictions in CASR for cyclist left turn indications (cropped from the original images). Remind that we are using a vehicle-centric criteria, this is why for oncoming cyclist an indication as right-turn must be classified as left-turn. Fig. 11: Correct predictions in CASR for cyclist right turn indications (cropped from the original images). Remind that we are using a vehicle-centric criteria, this is why for oncoming cyclist an indication as left-turn must be classified as right-turn. Fig. 12: Examples of correct predictions in CASR for cyclist stop indications (cropped from the original images). Fig. 13: Examples of correct predictions in YouTube images (cropped from the original images). Fig. 14: Wrong predictions in CASR for cyclist indications (cropped from the original images). ’N.S.’ stands for no sign. Fig. 15: Wrong predictions in YouTube images (cropped from the original images). ’N.S.’ stands for no sign. 10 VI. CONCLUSION [19] F. Flohr, M. Dumitru-Guzu, J. Kooij, and D. Gavrila, “A probabilistic framework for joint pedestrian head and body orientation estimation,” We have evaluated a monocular vision-based pipeline for IEEE Trans. on Intelligent Transportation Systems, vol. 16, no. 4, pp. recognition of VRU intentions. We have addressed the pedes- 1872–1882, 2015. [20] A. Schulz and R. Stiefelhagen, “Pedestrian intention recognition using trian crossing/not-crossing problem in naturalistic driving con- latent-dynamic conditional random fields,” in IV, 2015. ditions (JAAD dataset). We have addressed the recognition of [21] F. Schneemann and P. Heinemann, “Context-based detection of pedes- cyclist arm signs, in this case elaborating our own dataset, trian crossing intention for autonomous driving in urban environments,” in IROS, 2016. CASR. In both cases, we have applied the same procedure [22] Z. Cao, T. Simon, S. Wei, and Y. Sheikh, “Realtime multi-person 2D since our work hypothesis is that human skeletons fitted on pose estimation using part affinity fields,” in CVPR, 2017. 2D images already convey core information for VRU intention [23] G. Gkioxari, B. Hariharan, R. Girshick, and J. Malik, “R-CNNs for pose estimation and action detection,” arXiv:1406.5212, 2014. recognition. The obtained results support this hypothesis, [24] D.Ridel, E. Rehder, M. Lauer, C. Stiller, and D. Wolf, “A literature since by analysing features of the the fitted skeletons over review on the prediction of pedestrian behavior in urban scenarios,” in a relatively small temporal sliding window (∼ 500ms), the ITSC, 2018. [25] J. Kooij, F. Flohr, E. Pool, and D. Gavrila, “Context-based path pre- recognition task is performed with a high accuracy. We have diction for targets with switching dynamics,” International Journal of showed quantitative results supporting this claim, and we have Computer Vision, vol. 127, no. 3, p. 239262, 2019. also brought qualitative results (correct recognition cases, cur- [26] R. Quintero, I. Parra, D. Fernández-Llorca, and M. Sotelo, “Pedestrian path, pose, and intention prediction through gaussian process dynamical rent failure cases, top-features driving recognition) illustrating models and pedestrian activity recognition,” IEEE Trans. on Intelligent the reasons. Other researchers can use our approach as part Transportation Systems, May 2018. of a modular perception pipeline, as affordances on end-to- [27] O. Ghori, R. Mackowiak, M. Bautista, N. Beuter, L. Drumond, F. Diego, and B. Ommer, “Learning to forecast pedestrian intention from pose end driving models [38], or as additional cue on systems dynamics,” in IV, 2018. performing 3D trajectory prediction [25]. [28] Q. Deng, R. Tian, Y. Chen, and K. Li, “Skeleton model based behavior recognition for pedestrians and cyclists from vehicle scene camera,” in IV, 2018. R EFERENCES [29] H. Kretzschmar and J. Zhu, “Cyclist hand signal detection by an autonomous vehicle,” US Patent 9,014,905, April 2015. [1] D. Gerónimo and A. López, Vision-based Pedestrian Protection Systems [30] K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask R-CNN,” in for Intelligent Vehicles. SpringerBriefs in Comp. Sc., 2014. ICCV, 2017. [31] R. Girshick, I. Radosavovic, G. Gkioxari, P. Dollár, and K. He, “Detec- [2] S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards real- tron,” https://github.com/facebookresearch/detectron, 2018. time object detection with region proposal networks,” in NIPS, 2015. [32] Y. Wu et al., “Tensorpack,” https://github.com/tensorpack/, 2016. [3] X. Li, F. Flohr, Y. Yang, H. Xiong, M. Braun, S. Pan, K. Li, and [33] L. Zheng, Z. Bie, Y. Sun, J. Wang, C. Su, S. Wang, and Q. Tian, “Mars: D. Gavrila, “A new benchmark for vision-based cyclist detection,” in A video benchmark for large-scale person re-identification,” in ECCV, IV, 2016. 2016. [4] S. Zhang, R. Benenson, M. Omran, J. Hosang, and B. Schiele, “Towards [34] T. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollar, reaching human performance in pedestrian detection,” IEEE Trans. on and C. Zitnick, “Microsoft COCO: Common objects in context,” in Pattern Analysis and Machine Intelligence, vol. 40, pp. 973–986, 2017. ECCV, 2014. [5] X. Li, L. Li, F. Flohr, J. Wang, X. Xiong, M. Bernhard, S. Pan, [35] X. Chen and A. Gupta, “An implementation of Faster RCNN with study D. Gavrila, and K. Li, “A unified framework for concurrent pedestrian for region sampling,” arXiv preprint arXiv:1702.02138, 2017. and cyclist detection,” IEEE Trans. on Intelligent Transportation Sys- [36] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, tems, vol. 18, no. 2, pp. 269–281, 2017. O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vander- [6] A.Eldesokey, M. Felsberg, and F. Khan, “Ellipse detection for visual plas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duch- cyclists analysis in the wild,” in CAIP, 2017. esnay, “Scikit-learn: Machine learning in Python,” Journal of Machine [7] N. Wojke, A. Bewley, and D. Paulus, “Simple online and realtime Learning Research, vol. 12, pp. 2825–2830, 2011. tracking with a deep association metric,” in ICIP, 2017. [37] H. Jhuang, J. Gall, S. Zuffi, C. Schmid, and M. Black, “Towards [8] F. Flohr, “Vulnerable road user detection and orientation estimation understanding action recognition,” in ICCV, 2013. for context-aware automated driving,” Ph.D. dissertation, Informatics [38] A. Sauer, N. Savinov, and A. Geiger, “Conditional affordance learning Institute, University of Amsterdam, October 2018. for driving in urban environments,” arXiv:1806.06498, 2018. [9] M. Meinecke, M. Obojski, D. Gavrila, E. Marc, R. Morris, M. Töns, and L. Lettelier, “Strategies in terms of vulnerable road users,” EU Project Zhijie Fang received his B.S. degree in Au- SAVE-U, Deliverable D6, 2003. tomation from South China Agricultural University, [10] Z. Fang, D. Vázquez, and A. López, “On-board detection of pedestrian Guangzhou, China, in 2011. He received his M.S. intentions,” Sensors, September 2017. degree in Control Science and Control Engineer- [11] ——, “Is the pedestrian going to cross? answering by 2d pose estima- ing from South China University of Technology, tion,” in IV, 2018. Guangzhou, China, in 2014. And he is currently [12] A. Rasouli, I. Kotseruba, and J. K. Tsotsos, “Agreeing to cross: How a Ph.D. Candidate in the Department of Computer drivers and pedestrians communicate,” in IV, 2017. Science at the Universitat Autònoma de Barcelona [13] ——, “Are they going to cross? a benchmark dataset and baseline for (UAB). His research interests are in the areas of pedestrian crosswalk behavior,” in ICCV, Workshop, 2017. computer vision and pedestrian behavior analyses. [14] N. Schneider and D. Gavrila, “Pedestrian path prediction with recursive Antonio M. López is the PI of the Autonomous bayesian filters: A comparative study,” in GCPR, 2013. Driving lab of the Computer Vision Center (CVC) [15] C. Keller and D. Gavrila, “Will the pedestrian cross? a study on at the UAB, where he has also a tenure position as pedestrian path prediction,” IEEE Trans. on Intelligent Transportation associated professor of the Comp. Sci. Dpt. Antonio Systems, vol. 15, no. 2, pp. 494–506, 2014. has a long research trajectory at the intersection [16] S. Köhler, M.Goldhammer, K. Zindler, K. Doll, and K. Dietmeyer, of computer vision, computer graphics, machine “Stereo-vision-based pedestrian’s intention detection in a moving ve- learning and autonomous driving. Antonio has been hicle,” in ITSC, 2015. deeply involved in the creation of the SYNTHIA [17] B. Völz, K. Behrendt, H. Mielenz, I. Gilitschenski, R. Siegwart, and dataset and the CARLA open-source simulator. He J. Nieto, “A data-driven approach for pedestrian intention estimation,” is actively working with industry partners to bring in ITSC, 2016. state-of-the-art techniques to the field of autonomous [18] E. Rehder, H. Kloeden, and C. Stiller, “Head detection and orientation driving. Antonio is granted by the Catalan ICREA Academia program. estimation for pedestrian safety,” in IV, 2014.