sensors Article Hierarchical Novelty Detection for Traffic Sign Recognition Idoia Ruiz * and Joan Serrat Computer Vision Center and Computer Science Department, Universitat Autònoma de Barcelona, 08193 Bellaterra, Spain; joans@cvc.uab.cat * Correspondence: iruiz@cvc.uab.cat; Tel.: +34-93-581-1828 Abstract: Recent works have made significant progress in novelty detection, i.e., the problem of detecting samples of novel classes, never seen during training, while classifying those that belong to known classes. However, the only information this task provides about novel samples is that they are unknown. In this work, we leverage hierarchical taxonomies of classes to provide informative outputs for samples of novel classes. We predict their closest class in the taxonomy, i.e., its parent class. We address this problem, known as hierarchical novelty detection, by proposing a novel loss, namely Hierarchical Cosine Loss that is designed to learn class prototypes along with an embedding of discriminative features consistent with the taxonomy. We apply it to traffic sign recognition, where we predict the parent class semantics for new types of traffic signs. Our model beats state-of-the art approaches on two large scale traffic sign benchmarks, Mapillary Traffic Sign Dataset (MTSD) and Tsinghua-Tencent 100K (TT100K), and performs similarly on natural images benchmarks (AWA2, CUB). For TT100K and MTSD, our approach is able to detect novel samples at the correct nodes of the hierarchy with 81% and 36% of accuracy, respectively, at 80% known class accuracy. Keywords: novelty detection; hierarchical classification; deep learning; traffic sign recognition; autonomous driving; computer vision   Citation: Ruiz, I.; Serrat, J. 1. Introduction Hierarchical Novelty Detection for Deep neural networks have demonstrated to achieve outstanding performance on Traffic Sign Recognition. Sensors 2022, image classification. However, the problem of detecting samples that do not belong to any 22, 4389. https://doi.org/10.3390/ class known by the model, i.e., novelty detection, remains unsolved. Two challenges of this s22124389 task are that, first, classification networks trained by cross-entropy tend to be overconfident Academic Editor: Radu Danescu about their predictions, meaning they will assign a known class to any input fed to the Received: 3 May 2022 network with very high confidence. The second difficulty is that, by definition, there Accepted: 8 June 2022 is no training data for what is novel. There have been some efforts in addressing such Published: 10 June 2022 problem [1–4], but the binary output of these approaches only determines whether the sample belongs to a known class or is unknown. A desirable feature of classifiers would Publisher’s Note: MDPI stays neutral be, besides providing a novel/known decision, to produce an approximate prediction with regard to jurisdictional claims in of the novel class by taking advantage of the knowledge of the already learned classes. published maps and institutional affil- In particular, we go beyond vanilla novelty detection and study how to perform such iations. enhanced novelty detection under the framework of a hierarchical taxonomy of classes. This problem is known as hierarchical novelty detection [5]. It aims at correctly classifying samples of known classes, while also allocating the novel samples to the most suitable node Copyright: © 2022 by the authors. of the hierarchy, i.e., their parent class. Figure 1 illustrates a simplified example. Let us Licensee MDPI, Basel, Switzerland. assume a model trained on traffic sign recognition that has learned to only recognize speed This article is an open access article limit traffic signs of 10, 20, 50, 90, and 120. If the system is fed a sample image of a 30-speed distributed under the terms and limit, it then should predict that this sample belongs to a novel class, and more precisely conditions of the Creative Commons that it is a speed limit sign. Attribution (CC BY) license (https:// This problem has been traditionally studied as two independent tasks in the literature, creativecommons.org/licenses/by/ i.e., novelty detection and hierarchical classification. Solving the joint task, however, has 4.0/). the advantage that novelty detection can benefit from the hierarchical taxonomy of classes. Sensors 2022, 22, 4389. https://doi.org/10.3390/s22124389 https://www.mdpi.com/journal/sensors Sensors 2022, 22, 4389 2 of 22 In academia, experiments are often restricted to certain sets of classes that compose the datasets. These data are limited and cannot comprise all the possible classes and variability of samples a real-life application faces. These experiments, moreover, are based on a closed- world assumption [6,7], i.e., systems consider the only existing classes are those seen at training time. Novelty detection, instead, necessarily considers an open world setting. By leveraging the hierarchical taxonomy of the known classes, we show it is possible to produce approximate predictions even for unknown samples, by classifying them to the closest concept in such semantic taxonomy. Figure 1. Example of hierarchical novelty detection on traffic sign recognition. Our system is trained to recognize speed limit traffic signs of 10, 20, 50, 90, and 120. When it is fed an image of a speed limit traffic sign of 30, it should predict that it belongs to a novel class (never seen during training), but also that it is a speed limit traffic sign, placing correctly the novel sample in the hierarchical taxonomy of known classes. In this work, we propose to solve the problem of hierarchical novelty detection by introducing a novel loss function, i.e., Hierarchical Cosine Loss (HCL), which learns an embedding of discriminative features that is consistent with the taxonomy class relation- ships by encoding taxonomy-based constraints. In this embedding, every known class that corresponds to either a parent or leaf node is represented by a prototype. Prototypes enable the classification of any kind of sample, including novel ones. HCL is based on a normalized version of the softmax loss reformulated from a cosine perspective. It op- timizes the cosine similarity at training time between features and corresponding class prototypes. Consistently, we perform the novelty decision at test time by using the same metric. By mapping the sample into the embedding space, our approach assigns the sample features to the prototype with the highest cosine similarity. To the best of our knowledge, there is only a previous work [5] that has approached this problem. The authors instead employ confidence calibrated classifiers [3] to overcome the difficulty of the overconfidence of models trained with standard softmax. Similar approaches to ours [8] have been proved to increase the performance on the face recognition task in comparison to the standard softmax formulation that the methods in [5] apply. Whereas the mathematical background is similar, there is a subtle conceptual difference among both formulations. While standard cross-entropy answers the question What is this sample? ours tries to find the response to What does this sample look like most? The latter setting seems to be more appropriate to classify unknown samples by finding the most similar known class. Sensors 2022, 22, 4389 3 of 22 Our solution can be a powerful tool for practical applications. For instance, a potential application is to ease annotation procedures that could be semi-automated by providing the closest known class even for novel samples. In addition, it could be a first step towards class incremental learning [2,9], where one could extend the model with the newly learned classes. As a concrete application, autonomous driving technologies can benefit from it, e.g., by detecting new object categories automatically in a navigation system and suggesting the most similar known class. The aforementioned class incremental setting could also be used to build adaptive models to the challenging changing environment that autonomous driving systems face. We specifically address the traffic sign recognition problem [10–13]. This task is of special interest because, in the case of traffic signs, the semantic taxonomy is strongly related to the visual appearance. The categories are human-built so that the meaning is intended to be visually represented. Therefore, one could build an adaptive traffic sign detector that is able to infer the meaning, at least partially, of the detected novel signs. In summary, the contributions of this work are the following: • A hierarchical novelty detection framework that is able to detect novel samples that belong to classes not seen during training, also placing them at the correct node of the taxonomy, i.e., predict the parent class. For this purpose, we introduce a novel loss function, i.e., Hierarchical Cosine Loss, which incorporates hierarchical constraints and optimizes the cosine similarity as the confidence metric, differently from most of current approaches that are based on class probabilities. • A specific application to traffic sign recognition. We introduce the taxonomies and appropriate splits for two large scale traffic signs datasets, Mapillary Traffic Sign Dataset (MTSD) and Tsinghua-Tencent 100K (TT100K). • We show that HCL significantly outperforms state-of-the-art approaches on these traffic sign benchmarks. For TT100K and MTSD, our method is able to detect novel samples from unknown classes at the correct nodes of the hierarchy with 75% and 24% accuracy when it correctly classifies known classes with 90% accuracy, respectively. It also reaches 81% and 36% novel accuracy at 80% known accuracy for TT100K and MTSD, respectively. Additionally, on the natural images datasets AWA2 and CUB, it achieves equivalent performance to state-of-the-art models. • A new hierarchical novelty detection metric, i.e., the average error distance d¯h , to eval- uate the errors produced under a hierarchical setting. It measures how far in the hierarchy we predict novel classes from the correct node. • An ablation study that analyzes the individual performance of the HCL terms, dis- cussing their benefits and drawbacks. 2. Related Work 2.1. Novelty Detection Broadly, novelty detection belongs to the field of study of out-of-distribution detection [1,2,14,15], that consists of identifying samples that do not belong to the dis- tribution of the training data (in-distribution). More specifically, novelty detection aims to classify known classes while detecting novel samples that correspond to classes never seen during training. For instance, the authors of [1] address out-of-distribution detection by proposing a metric learning based approach. They distinguish among novelties and anomalies depending on the resemblance w.r.t. the in-distribution data. Similarly to us, they apply it to traffic sign recognition. However, they only provide a binary output that classi- fies a sample into either a known class or a generic class of novelty. Our approach instead provides information about what kind of novelty it is, by predicting its parent class as the expected output. In a different direction, the work in [2] considers both out of distribution detection and adversarial attacks, as both problems consist of detecting abnormal samples. They propose a Gaussian discriminant analysis resulting in a confidence score based on the Mahalanobis distance. Moreover, they apply their approach into a class incremental Sensors 2022, 22, 4389 4 of 22 setting framework, showing they are able to incorporate new classes without retraining the models. There are no works other than [5] directly addressing hierarchical novelty detection. The reason is probably that it is a concrete and complex task that merges two problems traditionally studied separately, i.e., hierarchical classification and novelty detection. The authors of [5] introduced the problem for the first time and proposed two different models. The first Top-Down model trains confidence-calibrated classifiers [3], which, besides training the standard cross-entropy loss, minimize the KL divergence of the probability vector w.r.t. the uniform distribution. At test time, it makes top-down decisions so that at each node it measures the KL divergence to evaluate whether the classifier is confident about the prediction, which determines if the sample is novel or known when compared to a threshold. The second Flatten model trains the standard cross-entropy loss considering all classes, i.e., both leaf and super classes, and performs the decision ignoring the taxonomy. Additionally, they show the hierarchical embeddings can be employed to improve the performance on generalized zero shot learning. Both their proposed approaches employ the standard softmax objective and base their training and decision on class probabilities. Differently to them, we train our embeddings by optimizing the cosine similarity instead of the inner product and perform the novelty decision based on this similarity metric. Furthermore, our approach learns an embedding of discriminative features that is consistent with the taxonomy class relationships. Nevertheless, there exist some problems that are conceptually similar. One of the closest problems is zero-shot learning (ZSL) [16–19], where the goal is to classify samples of classes not seen during training. The base idea of hierarchical novelty detection, i.e., to use the knowledge of the known classes to recognize the novel ones, is shared with ZSL. It, however, requires additional information about the known classes to be given, in the form of attributes or text description transformed into embeddings, while hierarchical novelty detection only relies on the class taxonomy. 2.2. Hierarchical Classification Considering hierarchical class taxonomies in the classification problem has been widely studied in the literature [20–24]. The problem of hierarchical novelty detection actually comprises hierarchical classification of the known classes. In [20], the authors take advantage from hierarchical taxonomies of classes for error measurement. They propose two methods based on the cross-entropy loss that aim to minimize the asymmetric cost of the errors produced. Their error evaluation employs the height of the lowest common ancestor (LCA) among the predicted and the ground-truth classes in the taxonomy tree. This is similar to the metric we propose for the task of hierarchical novelty detection in Section 5.2.1, the average error distance d¯h . Differently, we use the distance in the tree between both classes that corresponds to the sum of distances from both the predicted and ground truth classes to their LCA. More recently, a prototypical network is introduced in [21] that is supervised by employing a cost matrix that encodes hierarchical relationships among classes, then penalizing large hierarchical errors. It is conceptually similar to our proposed loss in that they also incorporate hierarchical constraints to learn an optimal embedding. In addition, they consider the Average Hierarchical Cost as a metric to evaluate classification errors, which matches the definition of our average error distance d¯h metric, but in the context of hierarchical classification. Another related problem is long-tailed recognition that consists of correctly classifying classes from which many of them are underrepresented in the training data. This often matches a real-life scenario, where having balanced data for all the classes is unlikely. The obvious differences are that their classes are highly imbalanced but at least one sample per class is seen during training, and they do not need to make a novel/known class decision. Some works employ class hierarchies in their solution. For instance, the authors of [25] propose to solve this problem under a hierarchical class taxonomy framework, then providing from coarse to fine-grained predictions according to the confidence. This enables Sensors 2022, 22, 4389 5 of 22 the models to reject classification at different levels. More recently, the authors of [26] transforms the problem into a hierarchical classification one by building a tree whose levels correspond to different degrees of difficulty according to how imbalanced the data are, then transferring the knowledge across levels. 2.3. Cosine Losses There exist diverse works that, similarly to us, propose loss functions based on modify- ing the softmax loss from a cosine perspective to improve its performance. The softmax loss, in this context, refers to a cross entropy loss preceded by a softmax activation and a fully connected layer. These works are commonly applied to the face recognition task, where learning discriminative features is essential to distinguish identities. They also benefit from this loss formulation because they use the cosine similarity at test time. The first work that opened this line of research was [27], where, based on the softmax loss, the proposed L-softmax loss included a new angular margin hyperparameter that acts on the class decision boundaries to enforce inter-class variance and then push the discriminative power of the features. SphereFace [28] normalized the weights of the last fully-connected layer on the L-softmax loss, making them lie on a hyphersphere. A normal- ized version of the softmax loss was introduced in [8], where they normalized both features and class weights so that the only variable to be optimized is the cosine of the angle between them. Later, the authors of [29] added a margin parameter to it to increase the discrimina- tive power of features. This margin separates the decision boundary between classes in the embedding space, at the cost of introducing a new hyperparameter. In our work instead, to learn discriminative features under a hierarchical setting, we propose additional terms to the loss that encode hierarchy-based constraints, being consistent with the problem we aim to solve. Similarly to [29], the authors of [30] also introduced a margin hyperparameter but applied on the angle. Finally, in an effort to improve the aforementioned methods, AdaCos [31] proposes a hyperparameter-free approach, leveraging a dynamically adaptive scale parameter that is adjusted automatically. Simultaneously, RegularFace [32] proposed an exclusive regularization term to the loss to further push inter-class discriminability by optimizing angular distance among classes. 3. Hierarchical Novelty Detection In this section, we first describe the setting of the hierarchical novelty detection problem in Section 3.1 and then introduce our proposed Hierarchical Cosine Loss in Section 3.2. 3.1. Class Taxonomy In hierarchical novelty detection, the classes are organized by a hierarchy of known classes that is built based on their semantics. The resulting taxonomies of the datasets considered in this work are trees, where all nodes have at least two children classes and a single parent. As an example, Figure 1 shows a subset of the taxonomy of MTSD. The dataset is split into two sets of disjoint classes: known and novel. Known classes are used during training to learn an embedding, while novel classes are not included in the hierarchy; they are never seen during training and our goal is to predict the correct parent (known) class for the novel samples at test time. Datasets provide samples for known leaf classes. However, our approach also needs sets of samples that represent the parent classes. To this end, we employ a relabeling strategy as in [5]. We select a percentage of the samples of the leaf classes to be relabeled as their parent class. We refer to this percentage as the relabeling rate rrate . This procedure is recursively repeated in a bottom-up manner from the bottom nodes to their parents until we reach the root and all the nodes are assigned samples. The subset of samples is chosen randomly and is different for each epoch. Sensors 2022, 22, 4389 6 of 22 3.2. Hierarchical Cosine Loss We introduce the Hierarchical Cosine Loss (HCL) in order to learn an embedding for the known classes. HCL comprises a layer of learnable parameters that corresponds to a fully connected layer with no bias. The HCL layer is appended after the feature layer of a ResNet-101 backbone, which serves as a feature extractor. Our loss, HCL, is composed by a set of terms that enforce learning discriminative features, leveraging the class hierarchy. It is defined as follows: HCL = λ NS L NS + λ HC L HC + λCT LCT + λ HT L HT , (1) where L NS , L HC , LCT , and L HT stand for Normalized Softmax, Hierarchical Centers, C- triplet, and Hierarchical Triplet loss, respectively, and λ NS , λ HC , λCT , and λ HT are their regularization parameters. Normalized Softmax Loss L NS . A reformulation of the softmax loss was introduced in [8], consisting of applying normalization on both the weights from the last fully- connected layer, whose bias is set to 0, and the feature vectors. This results in optimizing the cosine similarity instead of the inner product. We refer to this loss as Normalized Softmax Loss (NSL) L NS . NSL is defined as 1 es cos(θyi ,i ) L NS = N ∑ −log s cos(θ j,i ) , (2) i ∑j e where yi is the ground truth label of the i-th sample, N is the number of samples and θ j,i is the angle between Wj and xi , with Wj being a weight vector of the fully-connected layer for the j-th class and xi the feature vector of the i-th sample. A weight vector Wj can be interpreted as a representative vector of the j-th class, and we refer to it as a class prototype. For a class whose features are properly separated in the embedding space, its prototype would correspond to the mean of the features. By applying L2 normalization, we fix kWj k = 1 and k x k = s. This results in optimizing only the cosine of the angle, as the norms will not contribute to the loss. After normalization, the feature vectors lie on a hypersphere, where the scaling parameter s controls its radius and the resulting features are separable in the angular space, reducing intra-class angular variability and pushing inter-class variance within the hypersphere. This consequently enforces removing radial variations. C-Triplet loss LCT . This re-formulation of the softmax loss can also be translated into the Contrastive or Triplet losses, which inspired us to propose the following loss terms that incorporate hierarchical constraints. In [8], the authors introduce the C-triplet loss LT 0 as the modified version of the triplet loss that is defined as follows: LT 0 = max(0, m + k x̃i − Wj k22 − k x̃i − Wk k22 ), ∀yi = j, yi 6= k, (3) where x̃ = xs and m is a margin parameter. Note that both x̃ and Wj are normalized, then we could re-formulate it in terms of the cosine similarity. Considering that k x̃i − Wj k22 = 2 − 2WjT x̃i and WjT x̃i = cos θ j,i , LT 0 can also be defined as LT 0 = max(0, m + 2 cos θk,i − 2 cos θ j,i ), ∀yi = j, yi 6= k. (4) Then, considering pairs of different classes i, j, we define our C-triplet loss term LCT as LCTi = max(0, cos θ j,i − cos θi,i + mCT ), ∀yi 6= j. (5) where the margin parameter mCT is set to zero in all our experiments for simplicity. This term is intended to increase the discriminative power of the features, increasing the inter- class variance. It encodes that the features of a class should be closer to their class than to other class centers, i.e., the cosine similarity is higher among the features of a class xi and its prototype Wyi than to the prototypes of different classes Wj | yi 6= j. Sensors 2022, 22, 4389 7 of 22 Hierarchical Triplet loss L HT . To further enforce discriminative features based on the hierarchical relationships, we propose the Hierarchical Triplet term L HT that is defined as L HTi = max(0, cos θk,i − cos θ j,i + m HT ), ∀i, j, k | yi 6= j 6= k, dh (yi , j) < dh (yi , k), (6) where dh is the hierarchical distance between two nodes in the taxonomy, and we refer the reader to Section 5.2.1 for more details on this distance. m HT is a margin parameter and is set to zero in all our experiments. The purpose of this term is that features of a class should be closer to the prototypes of those classes that are closer in the taxonomy. For instance, a speed limit traffic sign class will be closer to any other speed limit sign than to any direction traffic sign. Figure 2 illustrates an example. The effect of this term is then to distribute the features in the hypersphere according to the taxonomical relationships. Figure 2. Interpretation of Hierarchical Triplet loss term L HT . Samples are forced to be closer to prototypes of classes that are closer in the taxonomy. In this example, the anchor sample is a speed limit sign of 30, while the positive class is the speed limit sign of 90, which is closer in the taxonomy than the direction sign class that is the negative class of the triplet. Hierarchical Centers loss L HC . Similarly to the Hierarchical Triplets term L HT , the Hi- erarchical Centers loss L HC aims to increase the separation in the angular space of the class prototypes Wj based on the hierarchical relationships between classes. The difference is that, instead of being applied to the distance among features and prototypes, it only affects the class prototypes Wj . Thus, this term enforces a higher similarity among class prototypes that are closer in the taxonomy. L HC is defined as LCi = max(0, cos φyi ,k − cos φyi ,j + mC ), ∀i, j, k | yi 6= j 6= k, dh (yi , j) < dh (yi , k), (7) where φyi ,j is the angle between Wyi and Wj . mC is a margin parameter and is set to 0.05 in all our experiments. 3.3. Inference By training HCL for both known leaf and super classes, we learn the set of class prototypes, i.e., class weights Wj from the last fully connected layer that identifies all the known classes. These, at test time, can be compared against the features of the test samples to perform classification. At inference time, for every test sample, we compute the cosine similarity between its features and all the class prototypes Wj . These features are extracted from the ResNet-101 model as we do at training time. We add an offset to the cosine similarities of the super Sensors 2022, 22, 4389 8 of 22 classes, which controls the trade-off between known and novel class accuracies. Its value can be varied within a range to select the desired working point. This is needed to compute the metrics detailed in Section 5.2. The test samples are finally classified to the class whose prototype Wj has the highest similarity w.r.t. their features, after applying the offset. Then, if a sample is assigned a leaf class, it means it corresponds to a sample of this known leaf class, while if the sample is assigned a super class, it is considered as a novelty under this parent class. For instance, a sample classified as a regulatory traffic sign is a sign of an unknown class of type regulatory that in the taxonomy would be a child class of regulatory. Note that, differently from other hierarchical classification methods [5], we do not follow a top-down strategy. This avoids top-down error aggregation that happens when the prediction at the top-most levels is wrong, and is magnified with complex and deep taxonomies. Therefore, we do the classification at inference time not considering any class taxonomy, all the classes being equally probable. 4. Datasets We consider two kinds of datasets to assess the performance of our approach. First, to compare it against the state-of-the-art methods of hierarchical novelty detection, we em- ploy the evaluation setting proposed in [5] as well as the datasets, CUB [33] and AWA2 [16]. Additionally, because we aim to apply our method on a traffic sign recognition framework, we choose two large scale traffic sign benchmarks: Tsinghua-Tencent 100K (TT100K) [34] and Mapillary Traffic Sign Dataset (MTSD) [35]. The original classes of these datasets are split into known and novel. Those that are known correspond to the leaf classes of the hierarchy. Among the samples of the known leaf classes, we build train, validation, and test splits. Train samples are used to train the model, validation for hyperparameter optimization, and test samples are used to evaluate the classification accuracy on the known classes. The details on how we make the splits are detailed in the following sections for each dataset. The data to reproduce our experiments are available in [36]. A dataset is more challenging as it has a larger number of samples, categories, and has a more complex taxonomy of classes [37]. Table 1 contains these data for the datasets evaluated in this paper. Note this information corresponds to the samples used in our experiments, where we have discarded some of the samples, and may differ from origi- nal benchmark statistics. Finally, the class taxonomies for these benchmarks are shown graphically in Appendix B. Table 1. Datasets overview: number of samples, parent and leaf classes in the taxonomy tree and its height, for both known and novel splits. The height of a tree is the height of its root class, so that a tree of two levels is of height 1. Known Novel Dataset # Samples # Parents # Leaves Height # Classes # Samples AWA2 29,408 21 40 5 10 7913 CUB 8814 43 150 5 50 2966 TT100K 21,956 14 80 2 23 1735 MTSD 65,312 40 164 3 39 4743 4.1. Tsinghua-Tencent 100K (TT100K) The Tsinghua-Tencent 100K [34] dataset is one of the first large scale traffic sign benchmarks. It contains samples under different illuminance and weather conditions, extracted from real-life street view panoramas. In our experiments, we have used the cropped images of traffic signs. A difficulty of this dataset is that it is highly imbalanced. It is built from real-life images, where different traffic signs do not appear with the same frequency. Only 45 classes out of 221 have more than 100 examples, while the largest class has 2819 samples. Sensors 2022, 22, 4389 9 of 22 The criteria to decide which classes belong to the novel split are based on the number of samples per class. We first discard the classes with less than 10 samples to avoid errors. From the remaining classes, we take 20% of those least populated as novel, regardless of their position in the taxonomy. We think this split is the one that best simulates the data in a real-world application, i.e., for a novel class, the goal is to correctly classify its samples in the taxonomy, not needing many of them, while known classes should be properly learnt from a larger number of samples. The most logical option is therefore to select the classes with fewer samples as novel. To build the train/test splits, for each known class, we keep 20% of the samples for tests and, within the remaining samples, 20% are used for validation. Since this dataset has not been previously used for hierarchical novelty detection, we have built a taxonomy based on class semantics, e.g., for traffic signs of prohibition limit of 20, they should have a parent class that comprises prohibition limit signs at other speeds, while this class should have a parent class that comprises any kind of prohibition sign as well. A visual representation of the built taxonomy is shown in Appendix B. 4.2. Mapillary Traffic Sign Dataset (MTSD) Recently, the Mapillary Traffic Sign Dataset has been introduced in [35]. It is the largest and most diverse traffic sign benchmark up to date. While TT100K contains only standard circular and triangular shaped signs, MTSD also includes direction, information or highway signs. Moreover, the images have been captured by multiple different camera devices all over the world. The benchmark provides fully and partially annotated traffic signs, although our experiments are restricted to only the fully annotated samples. Similarly to TT100K, MTSD is imbalanced, despite having a larger number of samples per class. The original class taxonomy of MTSD distinguishes as independent classes those that contain templates with the same semantics and similar appearance. However, we consider semantic based taxonomies, i.e., our application intends to classify the samples according to their meaning and not their appearance. For consistency, we choose to merge different groups of templates that share the same semantics but are different in terms of appearance, into a single class, as shown in Figure 3. This increases the intra-class variability, but in exchange simplifies the taxonomy we would have if we distinguished these classes. From 313 classes in the original taxonomy, after merging those classes with the same semantics, the resulting taxonomy has 203 leaf classes. Among these, 74 classes have less than 100 samples, while the largest class has 2775 samples. Figure 3. Two examples of samples of MTSD that are distinguished as disjoint classes in the original benchmark because they share the same semantics but have a different appearance. Aiming at classifying according to semantics, we merge them as single classes. In the left case, we merge five classes into one (complementary–chevron-left) and, in the right case, we merge four classes into one (regulatory–no-parking). To build the hierarchical taxonomy, we create super classes that encompass the traffic sign categories provided in MTSD that share similar semantics, e.g., the classes regulatory– no-left-turn and regulatory–no-right-turn have a parent class regulatory–no-turn that at the same time will have a regulatory parent class that comprises all the regulatory signs. Note that, due to the different composition of classes of TT100K and MTSD, they do not share a unified traffic sign taxonomy. There is no universal traffic sign taxonomy, to the best of our knowledge. It would be an interesting objective to explore in future work or a practical application, however. Finally, we make the novel/known and train/test/validation splits by employing the same criteria and percentages as for TT100K. Sensors 2022, 22, 4389 10 of 22 4.3. AWA2, CUB We employ the taxonomies of AWA2 and CUB provided in [5], which are built from the WordNet hierarchy by using its hypernym–hyponym relationships. Visual representations of the resulting hierarchies can be found in Appendix B. An interesting aspect of these taxonomies, differently to traffic signs, is that they obey to semantic hierarchical categories such as: placental mammal → carnivore → canine → dog → shepherd dog, where carnivore contains children classes as diverse as bear or feline. These high-level categories have no clear common features based on visual appearance. It is probably harder to learn how a carnivore looks than how a prohibition sign looks, since the concept is not reflected in the appearance but in deeper knowledge about what being a carnivore involves. These broad concepts are translated into a larger variability of samples under such conceptually high-level classes as well. 5. Evaluation 5.1. Experimental Setup We compare our method against the state-of-the-art models proposed in [5]. They propose three models, from which we consider TD+LOO and Relabel for the sake of a fair comparison. TD+LOO is their best performing model, while Relabel uses the same relabel strategy as ours to assign samples to parent classes. We run the implementation of these models provided by the authors. To train our model, we consider two settings depending on the experiment. The first one consists of using fixed precomputed features by freezing the weights of the ResNet-101 backbone, while training the HCL fully connected layer to learn the prototypes Wj of the classes, as detailed in Section 3.2. In this setting, we train HCL but not the ResNet-101 backbone. In the second setting instead, we train jointly all the layers of ResNet-101 and HCL. While carrying out the experiments, we noticed there was a moderate variability in the results even when using the same set of hyperparameters. Therefore, we repeat several times each experiment for a fixed set of hyperparameters. Instead of reporting the best- performing experiment from a set, the variability of the method is worth being analyzed. As we shall optimize our model to the validation set, a method whose performance is highly variable is not reliable because we do not know how it will perform on the test set. All our experiments are run on a set of GeForce GTX 1080, using multiple devices (at most four) in parallel when necessary, depending on the batch size. 5.2. Metrics In order to assess the performance of our method on hierarchical novelty detection, we consider the following metrics. For comparison against the state-of-the-art approach proposed in [5], we employ the AUC of the novel/known accuracy curve and the novel accuracy at a fixed known accuracy point. In their work, they select the point of 50% known accuracy as a reference. We use the average top-1 accuracy, so that a correct prediction is defined as follows: depending on the split. For known classes, their correct prediction is the ground truth label, while for novel classes, a correct prediction involves classifying it as the closest class in the taxonomy, i.e., its parent known class. The accuracy is averaged by the number of samples, independently of their label. The novel/known accuracy curve is obtained by adding an offset to the similarity metrics of the potential novel classes, i.e., parent nodes. This offset value is varied so that we increase/decrease novel accuracy in detriment/favor of known accuracy, as both splits hold a trade-off relationship. The novel/known accuracy curve is built from a range of offset values that allows for exploring all the available accuracy ranges. Accordingly, the AUC value is independent from the offset, i.e., it is independent from the working point. On traffic sign benchmarks, we additionally consider points of interest at higher known accuracy points. In this context, we are interested in a working point in which our system classifies correctly most of the known classes, while performing as best as possible Sensors 2022, 22, 4389 11 of 22 on the unknown ones. For this reason, the metrics that are more relevant are those of higher known accuracy points. In particular, we report the novel accuracy at 70% and 80% known accuracies, although we are interested in the range of known accuracy over 70%. For this reason, the AUC value is not a highly representative metric in our analysis, as it corresponds to the area for the full range. 5.2.1. Hierarchical Error Distance The accuracy only evaluates if the prediction matches the correct label but does not provide a measurement of the errors made. Specially under a hierarchical setting, we find this metric to be insufficient. Two wrong predictions of different degree of importance are treated as equally wrong by the accuracy. For instance, if the true class of a sample is a 20 maximum speed limit regulatory sign, predicting its class as a 10 speed limit regulatory sign should be considered a smaller error than predicting it as a chevron left complementary sign. In fact, other works on hierarchical image classification [20,21] stress the importance of optimizing error based metrics besides accuracy. As a complementary metric to the accuracy, we introduce the hierarchical average error distance d¯h . It corresponds to the distance between the predicted and the correct class in the taxonomy tree. For the i-th sample, the hierarchical error distance dh ( pi , yi ) between the predicted class pi and its ground truth label yi is defined as the length of the shortest path in the tree that corresponds to the sum of distances from both classes pi and yi to their lowest common ancestor (LCA). The average error distance d¯h is then defined as 1 d¯h = N ∑ d h ( p i , y i ), (8) i where N is the total number of samples. Note that this distance metric is not normalized by the height of the taxonomy tree, which affects its maximum value, e.g., in a taxonomy of five levels, the maximum error distance is 10, while, in a taxonomy of 2 levels, it is 4. In our experiments, we report the hierarchical average error distance for the novel split only, to analyze its dependency w.r.t. the accuracy of the known split. This provides a measurement of the novelty detection error under such hierarchical setting. 6. Results and Discussion Our experiments are divided into three parts. To evaluate the performance of our approach, HCL, in Section 6.1, we compare it to the state-of-the-art models in hierarchical novelty detection, i.e., TD+LOO [5] and Relabel [5]. In Section 6.1.1, we first consider the benchmarks where these models were originally evaluated, i.e., AWA2 and CUB. Then, in Section 6.1.2, we perform the evaluation on the target traffic signs benchmarks TT100K and MTSD. In the next sections, we provide a more exhaustive evaluation of HCL on TT100K and MTSD. We compare the performance of different training strategies in Section 6.2. Finally, in Section 6.3, we analyze the individual contribution of each of the terms of HCL. 6.1. Comparison to the State of the Art 6.1.1. AWA2 and CUB For these experiments, we train HCL, TD+LOO and Relabel on top of features extracted from a ResNet-101 model that is only trained on ImageNet. This is the setting the authors of [5] chose, in their case claiming speed reasons. We use the exact hyperparameters and setting indicated by the authors. For HCL, the hyperparameters are chosen by optimizing them to the validation set (see Appendix A for details on hyperparameters). We report in Table 2 the metrics introduced in Section 5.2 comparing the performance of our approach to TD+LOO and Relabel. The values of the metrics correspond to the average of 50 experiments, and we provide an error of ±2σ. Figure 4 shows the novel/known accuracy trade-off and the average hierarchical error distance on the novel split over the Sensors 2022, 22, 4389 12 of 22 known split accuracy. The dark curves of the plots correspond to the average of the set of 50 repeated experiments, while the shaded area around illustrates ±2σ for each point. (a) (b) Figure 4. Novel/known accuracy trade-off and novel average hierarchical error distance over known accuracy, for HCL (red) and the state-of-the-art models TD+LOO [5] (green) and Relabel [5] (blue) for (a) AWA2 and (b) CUB. Table 2. Comparison of HCL against TD+LOO [5] and Relabel [5] on AWA2 and CUB. Performance is measured by the novel/known accuracy AUC and the novel accuracy and average hierarchi- cal error distance d¯h at 50% known accuracy. The reported values are the average from a set of 50 experiments ±2σ. Best metrics are highlighted in bold. AWA2 CUB Method AUC Novel acc Novel d¯h AUC Novel acc Novel d¯h @50% ↑ @50% ↓ @50% ↑ @50% ↓ TD+LOO [5] 25.7 ± 4.3 33.1 ± 6.1 1.82 ± 0.23 18.0 ± 1.0 9.9 ± 1.1 2.56 ± 0.10 Relabel [5] 33.7 ± 5.9 38.7 ± 6.8 1.65 ± 0.12 28.9 ± 2.2 38.1 ± 3.5 1.56 ± 0.07 HCL 32.8 ± 1.8 36.4 ± 2.2 1.95 ± 0.05 27.6 ± 0.6 35.7 ± 1.2 1.34 ± 0.03 AWA2. Considering only the mean of the experiments, Relabel [5] is superior in terms of accuracy in the range up to 70% known accuracy, while HCL performs better in the highest known accuracy range. This is the reason why Relabel obtains the highest AUC. However, if we take into account the variability of the methods, HCL and Relabel perform very similarly, i.e., their curves overlap except for the highest known accuracy range. Regarding the novel hierarchical error distance d¯h , Relabel [5] consistently makes smaller errors on the novel split. CUB. In terms of accuracy, both Relabel and HCL perform similarly although the variability of Relabel is higher. Relabel performs better than the other variants up to ∼60% Sensors 2022, 22, 4389 13 of 22 known accuracy while HCL is superior in the uppermost known accuracy range. HCL makes a consistent smaller average error through all the accuracy ranges. Our results show HCL performs similarly to the state-of-the-art methods [5] on the natural images benchmarks, AWA2 and CUB. It shows a slightly higher novel accuracy at the highest known accuracy ranges, while the errors made by the model are smaller than TD+LOO and Relabel on CUB but higher on AWA2. 6.1.2. TT100K and MTSD Instead of using the features from a model trained on ImageNet as in the previous experiments, for TT100K and MTSD, we find it necessary to perform a fine-tuning of ResNet-101 using the cross-entropy loss. This is because traffic signs are a very specific kind of data, with a visual appearance different to ImageNet images. The comparison of performance when using features fine-tuned or not to the target dataset will be discussed later in Section 6.2. The fine-tuning is performed by training ResNet-101 for 1000 epochs using a batch size of 140 and a learning rate of 1 × 10−4 with an Adam optimizer for both datasets. Once ResNet-101 is trained, we extract the features to train TD+LOO, Relabel and our model, as we did for AWA and CUB. It is also possible to train simultaneously the ResNet-101 backbone and HCL. However, we chose to do a separate fine-tuning to keep the setting proposed in [5] for the sake of a fair comparison. The fine-tuning was performed only once, while the experiments for HCL, TD+LOO, and Relabel were repeated 50 times with the set of best performing hyperparameters on the validation set. We refer the reader to Appendix A for details on hyperparameters. We compare in Table 3 HCL to TD+LOO and Relabel. The reported metrics are the average value ±2σ from the set of 50 experiments. Figure 5 shows the novel/known accuracy trade-off and the average hierarchical error distance on the novel split over the known split accuracy. On both datasets, HCL consistently outperforms Relabel [5] and TD+LOO [5] by a large margin through all the ranges of accuracy both in terms of accuracy and average novel hierarchical error distance. Our results suggest that our approach is more suitable for traffic signs’ datasets. A possible explanation is related to the taxonomy of these datasets. Both TD+LOO and Relabel solely rely on the cross-entropy loss, but HCL learns an embedding of discriminative features that could benefit from taxonomies related to the visual appearance of the classes, e.g., prohibition signs have common visual features, while carnivore images do not have an indistinguishable visual feature. Table 3. Comparison of HCL against the state-of-the-art models on TT100K and MTSD. Performance is measured by the novel/known accuracy AUC and the novel accuracy and average hierarchical error distance d¯h at 50% and 70% known accuracies. The values are the average from a set of 50 experiments ±2σ. Best metrics are highlighted in bold. Novel acc ↑ Novel d¯h ↓ Dataset Method AUC @50% @70% @50% @70% TD+LOO [5] 42.2 ± 2.6 55.8 ± 4.6 12.6 ± 1.5 0.68 ± 0.09 1.47±0.20 TT100K Relabel [5] 48.4 ± 3.7 52.3 ± 4.0 48.7 ± 4.2 0.63 ± 0.06 0.66 ± 0.07 HCL 84.1 ± 0.7 87.2 ± 0.8 83.7 ± 1.1 0.15 ± 0.01 0.18 ± 0.01 TD+LOO [5] 30.6 ± 1.5 36.4 ± 2.6 9.5 ± 1.9 1.31 ± 0.10 2.12 ± 0.08 MTSD Relabel [5] 27.3 ± 3.8 30.9 ± 5.0 24.6 ± 3.6 1.28 ± 0.11 1.44 ± 0.09 HCL 44.2 ± 1.6 47.7 ± 2.1 40.5 ± 2.3 0.78 ± 0.04 0.89 ± 0.05 Sensors 2022, 22, 4389 14 of 22 (a) (b) Figure 5. Novel/known accuracy trade-off and novel average hierarchical error distance over known accuracy, for HCL (red) and the state-of-the-art models TD+LOO (green) and Relabel (blue) for (a) TT100K and (b) MTSD. 6.2. Training Strategies We consider three different settings to train HCL. In particular, we train HCL on top of features that are extracted from ResNet-101 models that are previously trained, either on only ImageNet, or fine-tuned to the target dataset via the cross-entropy loss. The third setting we compare is when we train simultaneously HCL and the ResNet-101 backbone that is pretrained on ImageNet. We keep the fine-tuning procedure that is detailed in the previous Section 6.1.2. Using the hyperparameters from Table A1, we repeat each new experiment for 10 times, as some training variants we compare in this section are time-consuming, and HCL was shown to be not so variable in previous results. Table 4 reports the average metrics ±2σ for these three training strategies at 50%, 70%, and 80% known accuracy points, while the performance through the entire range is depicted in Figure 6. The gap of performance on both datasets between using features from a network only trained on Imagenet, and features fine-tuned to the target dataset, justifies the need of performing such fine-tuning, especially because, in a traffic sign recognition application, we aim to maximize the novel accuracy at the highest known accuracy range. Sensors 2022, 22, 4389 15 of 22 Table 4. Comparison of different training strategies for HCL on MTSD and TT100K. I stands for ImageNet features, F for fine-tuned features and B for training both the backbone and HCL simulta- neously. We report the novel/known accuracy AUC and the novel accuracy and average hierarchical error distance d¯h at 50%, 70%, and 80% known accuracy points. Best metrics are highlighted in bold. AUC Novel acc ±2σ ↑ Novel d¯h ±2σ ↓ ±2σ @50% @70% @80% @50% @70% @80% TT100K I 54.0 ± 3.2 63.2 ± 2.8 45.3 ± 4.5 27.4 ± 4.6 0.43 ± 0.03 0.69 ± 0.02 0.95 ± 0.02 F 84.1 ± 0.7 87.2 ± 0.8 83.7 ± 1.1 80.7 ± 1.0 0.15 ± 0.01 0.18 ± 0.01 0.22 ± 0.01 B 71.4 ± 4.1 80.0 ± 7.8 60.6 ± 9.3 50.9 ± 6.5 0.22 ± 0.08 0.42 ± 0.09 0.54 ± 0.06 MTSD I 25.9 ± 1.0 34.3 ± 1.4 00.0±0.0 00.0±0.0 1.36 ± 0.02 - - F 44.2 ± 1.6 47.7 ± 2.1 40.5 ± 2.3 35.8 ± 2.2 0.78 ± 0.04 0.89 ± 0.05 0.99 ± 0.05 B 43.1 ± 8.2 47.4 ± 11.3 36.7 ± 10.9 30.8 ± 7.1 1.01 ± 0.20 1.16 ± 0.22 1.26 ± 0.19 (a) (b) Figure 6. Novel/known accuracy trade-off and novel average hierarchical error distance over known accuracy, for different training strategies for HCL on (a) TT100K and (b) MTSD. I stands for ImageNet features, F for fine-tuned features, and B for training both the backbone and HCL simultaneously. On TT100K, training HCL on top of fine-tuned features works significantly better than training jointly HCL and the ResNet-101 backbone. This is probably because fine-tuning on TT100K is overfitting the dataset, which has few samples of a very specific kind of data (traffic signs). Then, training HCL from high quality features as a starting point is much easier than jointly learning suitable features along with proper class prototypes that are consistent with the taxonomy. However, MTSD is a much larger dataset with a greater number of classes with higher inter and intra-class variability, as discussed in Section 4.2. The gap of performance in Sensors 2022, 22, 4389 16 of 22 this case is therefore smaller. The accuracy curve overlaps for almost the entire range, although the gap in error distance is consistent. This means for a very similar number of correct predictions, and the errors of the wrong predictions are smaller when we use fine-tuned features, presumably because these fine-tuned features allow for doing a more precise classification of novel samples. The reason might be learning two objectives, i.e., class prototypes and suitable features, is a more challenging task than only learning the prototypes with a fixed set of features. This involves a less noisy signal to learn from, since features are not being updated during training. This effect that also occurred on TT100K at a smaller scale is magnified with larger datasets with high intra-class variability, as in this case. As expected, the variability of the experiments when we train the backbone is higher than when we only train HCL, for both datasets. This is due to the additional variability introduced by training ResNet-101. To conclude, our proposed approach, HCL, reaches its highest performance when trained from features fine-tuned to the target dataset. It is able to predict correctly 75% and 24% of novel samples, for TT100K and MTSD, respectively, when we predict known samples with 90% accuracy. It also predicts novel samples with an accuracy of 81% and 36% at 80% known accuracy for TT100K and MTSD, respectively. 6.3. Ablation Study of Hierarchical Cosine Loss In order to analyze the individual contribution of the terms of HCL (Equation (1)), we conduct the following ablation study. We take as a baseline the contribution of only the Normalized Softmax loss (NSL) L NS , then adding the contribution of the remaining terms that will be finally compared to an experiment in which all the terms contribute to the training. The latter is the best performing HCL experiment, according to the validation set. We train the terms of HCL over a set of constant features, fine-tuned to the target dataset. These fine-tuned features correspond to those used in the previous experiments. Note that this independent training of HCL allows for isolating the effect of the loss. Otherwise, training jointly the backbone and HCL would introduce a variability that would mask the actual variation of the individual loss terms. In Table 5, we assess the individual performance of the different terms of HCL, defined in Section 3.2, on features fine-tuned to MTSD or TT100K. For each dataset, the first row shows, as a baseline, the metrics when we set the HCL regularization parameters to {λ NS , λ HC , λCT , λ HT } = {1, 0, 0, 0}, i.e., we train using only the NSL L NS . In the experiments of the next rows, we keep λ NS = 1 and add the different terms on each experiment, e.g., the second row corresponds to {λ NS , λ HC , λCT , λ HT } = {1, 10, 0, 0} where only the NSL L NS and Hierarchical Centers term L HC are contributing to the training. Similarly, the third row corresponds to the experiments where we use only the L NS and LCT terms with regularization parameters {λ NS , λ HC , λCT , λ HT } = {1, 0, 1, 0}, and in the fourth row we train using L NS and L HT regularized by {λ NS , λ HC , λCT , λ HT } = {1, 0, 0, 0.1}. The last row, where {λ NS , λ HC , λCT , λ HT } = {1, 10, 1, 0.1}, reports the performance of the full version of HCL, including all the terms. Despite the NSL weight λ NS always being set to 1, we made sure its contribution to the loss was not leading the training, i.e., the loss that is being analyzed individually at each case is not being neglected and actually contributes to the training. That is, we made sure that the weights applied to the individual terms were appropriate to show the individual effect of the loss terms. Each training variant has been repeated 10 times, and we report an error of 2σ on Table 5. For TT100K, the first experiment, in which we only use the NSL L NS , obtains the best average metrics among the compared variants. However, the difference of performance is very small. In fact, if we take into account the variability of the experiments, we could consider all the variants to perform similarly. The cause of this result might be that the performance on this dataset is so good that it reaches a limit that is hard to surpass. Making small modifications on the loss is not translated into a significant change in performance. In Sensors 2022, 22, 4389 17 of 22 this scenario, making even very small improvements is not straightforward, and it would probably mean it is overfitting the dataset. Table 5. Ablation study of the HCL terms. Performance is measured by the novel/known accuracy AUC and the novel accuracy and average hierarchical error distance d¯h at 70% and 80% known accuracies. The metrics are the average of 10 experiments ±2σ. Best metrics are highlighted in bold. {λ NS , λ HC , AUC Novel acc ±2σ ↑ Novel d¯h ±2σ ↓ Losses λCT , λ HT } ±2σ @70% @80% @70% @80% TT100K L NS {1, 0, 0, 0} 84.1 ± 0.6 83.9 ± 0.8 80.9 ± 0.7 0.18 ± 0.01 0.21 ± 0.01 L NS , L HC {1, 10, 0, 0} 84.0 ± 0.4 83.7 ± 0.5 80.7 ± 0.7 0.19 ± 0.01 0.22 ± 0.01 L NS , LCT {1, 0, 1, 0} 83.8 ± 0.9 83.6 ± 1.3 80.6 ± 1.2 0.19 ± 0.01 0.22 ± 0.01 L NS , L HT {1, 0, 0, 0.1} 84.1 ± 0.7 83.8 ± 0.9 80.7 ± 0.8 0.18 ± 0.01 0.22 ± 0.01 HCL {1, 10, 1, 0.1} 84.1 ± 0.7 83.7 ± 1.1 80.7 ± 1.0 0.18 ± 0.01 0.22 ± 0.01 MTSD L NS {1, 0, 0, 0} 41.8 ± 0.5 37.6 ± 0.7 33.7 ± 0.7 0.93 ± 0.01 1.03 ± 0.01 L NS , L HC {1, 10, 0, 0} 42.8 ± 1.1 38.9 ± 1.4 34.7 ± 1.4 0.91 ± 0.03 1.01 ± 0.02 L NS , LCT {1, 0, 1, 0} 42.5 ± 2.2 38.5 ± 2.8 34.1 ± 2.8 0.94 ± 0.07 1.04 ± 0.07 L NS , L HT {1, 0, 0, 0.1} 43.7 ± 1.2 39.9 ± 1.7 35.7 ± 1.8 0.88 ± 0.03 0.97 ± 0.03 HCL {1, 10, 1, 0.1} 44.2 ± 1.6 40.5 ± 2.3 35.8 ± 2.2 0.89 ± 0.05 0.99 ± 0.05 The results on MTSD are more enlightening; it is a more challenging dataset, closer to a real-life scenario. Using the different terms of HCL always improves the average novel accuracy at 70% and 80% known accuracies w.r.t. the NSL baseline. The variant that performs best on these metrics is the full version of HCL. The distance error is also decreased for all the variants except for L NS , LCT that obtains equivalent performance. The best error distance is achieved by L NS , L HT , but as before, if we consider the variability of the results, the differences w.r.t. the full version of HCL are not significant. A remarkable outcome we can draw from these experiments is that the Hierarchical Triplets term L HT improves the average metrics at the cost of increasing the variability of the method. This is expected as this constraint introduces different information that depends on the training data. As discussed in Section 3.2, we make triplets from the batch that is fed to the network. If batches are different, so will the triplets also be. In the case of MTSD, which is a much larger dataset than TT100K, it is possible to make a much larger number of triplets that introduce different information, consequently affecting the training result. This is also applied to the C-triplet term LCT for the same reason. There are more available pairs of different classes in a larger dataset, therefore affecting the training outcome. It is also worth mentioning that the variability of L HC is expected to be low due to the kind of experiments we carry. Its cost depends on the angle between class prototypes. Using fixed pre-computed features that are not changing through the training only requires finding the class prototypes. Training the ResNet-101 backbone would also be translated into a higher variability for this term. In summary, this ablation study shows that the proposed HCL terms can help with improving the performance, as shown in MTSD results. On TT100K, they do not improve the performance of the NSL alone on average because it already reaches a very high value. Some of the HCL terms (L HT , LCT ) have shown to increase the potential performance at the cost of increasing the variability of the results. Triplet mining strategies might help to mitigate this issue. 7. Conclusions We have addressed the problem of hierarchical novelty detection, specifically focused on traffic sign recognition. It involves classification along with detection of novel classes, and consists of predicting not only that a sample belongs to a novel class (never seen during training), but also its closest position in a semantic hierarchy of known classes. We have introduced a novel loss function, Hierarchical Cosine Loss, that learns jointly an Sensors 2022, 22, 4389 18 of 22 embedding of discriminative features consistent with the class taxonomy, as well as prototype representations for both leaf and parent classes. HCL achieves equivalent results to state- of-the-art approaches on natural images benchmarks, AWA2 and CUB, and significantly outperforms them on traffic sign datasets. For the latter experiments, we have contributed taxonomies and corresponding training splits for TT100K and MTSD, two challenging large scale traffic signs benchmarks that simulate real data of a traffic sign recognition application. Our approach is able to detect novel samples from unknown classes at the correct nodes of the hierarchy with 75% and 24% accuracy when we classify known classes with 90% accuracy, for TT100K and MTSD, respectively. It also reaches 81% and 36% novel accuracy at 80% known accuracy, for TT100K and MTSD, respectively. Finally, we have contributed an ablation study that analyzes the individual performance of the HCL terms. As a future line of research, our model could be applied to class incremental learning. By adding the prototypes of the detected novel classes at the proper taxonomy locations, our model could be extended to recognize new classes. Author Contributions: Conceptualization, J.S. and I.R.; methodology, J.S. and I.R.; software, J.S. and I.R.; validation, J.S. and I.R.; formal analysis, I.R.; investigation, I.R.; resources, J.S.; data curation, J.S. and I.R.; writing—original draft preparation, I.R.; writing—review and editing, J.S.; visualization, I.R.; supervision, J.S.; project administration, J.S.; funding acquisition, J.S. All authors have read and agreed to the published version of the manuscript. Funding: The authors acknowledge the support received for this research from the Spanish Grant Ref. PID2020-115734RB-C21 funded by MCIN/AEI/10.13039/501100011033. Institutional Review Board Statement: Not applicable. Informed Consent Statement: Not applicable. Data Availability Statement: The necessary data to reproduce the experiments as well as the imple- mentation of the method proposed in this manuscript are available in a GitHub public repository [36]. Acknowledgments: The authors also acknowledge the support of the Generalitat de Catalunya CERCA Program as well as its ACCIO agency to CVC’s general activities. Conflicts of Interest: The authors declare no conflict of interest. Abbreviations The following abbreviations are used in this manuscript: HCL Hierarchical Cosine Loss NSL Normalized Softmax Loss MTSD Mapillary Traffic Sign Dataset TT100K Tsinghua-Tencent 100K Appendix A. Hyperparameters All the hyperparameters were tuned by optimizing them to the validation set. Note that, in hierarchical novelty detection, there is no split of novel samples used as validation to search for hyperparameters. This, as discussed in [5], makes the problem more challenging, as we can only optimize our model to the validation set of the known classes, but the novel classes will always remain unknown. Appendix A.1. State-of-the-Art Models For the experiments on CUB and AWA, we use the parameters provided by the authors [5]. They train in a full-batch manner using an Adam optimizer with an initial learning rate of 10−2 , and it decays at most two times when loss improvement is less than 2 compared to the last epoch. They apply L2 norm weight decay with parameter 10−2 . For MTSD and TT100K, we keep the same setting, except for the relabeling rate of the Relabel model that was set to 15% and 30%, respectively. Sensors 2022, 22, 4389 19 of 22 Appendix A.2. HCL For all the experiments on HCL, we use an Adam optimizer with a learning rate of 0.01 and the hyperparameters from Table A1. In the experiments where we jointly train the backbone and HCL, the Adam optimizer uses a learning rate of 10−4 for the ResNet-101 backbone, for both datasets. Table A1 contains: the regularization parameters for the HCL loss, {λ NS , λ HC , λCT , λ HT }, the batch size (BS), number of epochs (nepochs ) and relabeling rate rrate . On the experiments of HCL being trained on precomputed features, we employ a full-batch training. The s parameter from the Normalized Softmax loss is always set to 40. For the experiments in the ablation study, we keep the same hyperparameters as when we train the full version of HCL. Table A1. Hyperparameters to train HCL. Experiment {λ NS , λ HC , λCT , λ HT } BS nepochs rrate (%) AWA2 Imagenet feat. {1, 1, 1, 0.01} full-batch 1000 15 CUB Imagenet feat. {1, 1, 1, 0.01} full-batch 1000 15 TT100K Imagenet feat. {1, 1, 1, 0.01} full-batch 1000 30 Finetuned feat. {1, 10, 1, 0.1} full-batch 1000 30 Backbone + HCL {1, 10, 1, 10} 280 300 30 MTSD Imagenet feat. {1, 1, 1, 0.01} full-batch 1000 15 Finetuned feat. {1, 10, 1, 0.1} full-batch 1000 15 Backbone + HCL {1, 10, 1, 10} 280 300 15 Appendix B. Taxonomy Figures Taxonomies for TT100K, MTSD, AWA2, and CUB are depicted in Figures A1–A4, respectively. ph5 pr20 pr30 pr40 ph4.8 pr50 ph4.5 pr60 ph4.2 ph4 pr70 pr80 .3 ph3 pm ph3 pm 10 2.8 pm 15 2.2 ph pm 20 ph2 ph 30 pm 0 5 pl9 pm 80 50 pl pm 70 p1 pl 55 60 p14 pl 0 p19 pl5 p_remove_limit pl5 p23 p_max_height 0 p5 p_max_weight pl4 5 p12 pl3 0 p16 pl3 p_arrow pl20 p6 pl15 p17 p_speed_limit p_bike pl120 pa10 pl110 pa13 n itio pl100 p_axis_len hib pa14 gth pro pl10 p10 io p11 ip p18 information i4 p2 i2 p22 p25 warning i_arrow p26 p27 i5 p3 i13 p8 i_speed_limit i12 p9 pb i10 pn e pn pg il90 po ps 3.2 wo il80 w6 w_crossing w5 pw 3 w5 il70 7 w45 w34 w_arrow 5 w32 w30 w47 w24 il60 w18 w3 il50 il1 10 il1 00 3 w1 0 w2 w21 w22 w15 w16 w42 Figure A1. TT100K class taxonomy. It contains both novel and known classes. See at full size in [36]. Sensors 2022, 22, 4389 20 of 22 regulatory--go regulatory-- go-straight-o -straight-or-tu re tory-- gu tor -no- -no s-vehic g y- y-good kin les r-turn-right a ry s-carr --n righ n-re n re lato ulat -by-hea vert la ehic or -no rn-o -tur rn-left regulatory--go-straigh d gu reg aking o-o -u o- g-da left -turn v regulatory--buses-only regulatory--bicycles-only regulatory--mopeds-and-bicycles-only regulatory--pedestrians-o regulatory--u-turn de rous-g rn --n y--turn-right ahead regulatory eft regulator one-way-righight t -ahead pe nge -tu regulat one ria ods regulat --dual-lan -go-straig ory tion tu regula ulatory--turn-l regu y-road gu ry or o re g regu tory--k ad-clo d-to-ve d-pede icycle regu latory-- --give-weight- ontrold-on-gtions ad d-limit-z ns - reg latory-- imum-s o-parkin gula lat it-70 -turn-right- rohibi rt reg ve --n yin o- latory ht-th only s reg ory--en y--turn-left ory-- o-o reg arking --no-s rough u icle s ory--o - reg tory--sh red-p tory--d tory end-of- peed-lim riorit ula y--one-w lato - g eg st --g --n y eh reg les- ula in -of-p y-- gulatory--no-p topping regulator le ula y tory reg -v of-spee yc p e r ula g o-straight ry-- ep-rig ed -n -of-p ds tory ad-c -bicyc rians- top rc bicyc s --n arkin ula ne-w -go-straig -on-right ula tory ared-p th-ped a ual-l ert ntry oto cle s reg d-of-m end-of gu no-v lat oo pt-m regu latory--end l reg regulatory- tory kee -g reg --ro le -s cy --n ulato traig --ro --end tor -e e ay-left anes y re or use s e c --sh o ay-stra d- av x ak es n o p-l e s-e he hicle t -or- -- -s a r o m no-b ycle ry ax - tory latory eft to --no o nly e -n tor-v - ry los ath - o -n ry-- -bic rn ht t la regu a to lato regula ry s ov t-or gu e o to regulat y- ula re -m -tu tory reg or ato -no la --no o-p en - ula gu ht-on- ht no les- -turn lat ul ry- est u re ry ry-- re ulato o g o an u hic -n eg re ulat g left regu re ng r les rians re tory--do al-pat ay-to- it an ula - regu tor regulatory-- way hi g ry regulatory-only d-b re regu reg et st to ula --turn regu tory-- ared-p m -of reg ula o latory s reg -s latory o lato u 0 -end gu g --n regu latory -9 la it --dua la regu lato ry-- ft-tu g-r ane er- re lato -not-bloh-bicyclescomin im -80 re ulato ry-- park sing -eit ry d-l regulatory ry-- ad warn gu it re ulato ry-- pas s-o ee to im g 0 -sp eed-l re ry-- ck-inte -and-pg-traf du --one- straight-on in l-l it-7 g warn g--do la re ula tory kee ry-- n rn-y estr -ah sid y- um anes -lim -60 gu p gu re ro s uble xim m-s eed h e-c ie ic e e la go g la ing-- it h ectio estri to re la ma imu m-sp ed-lim it-5 lator re re le -c gu p gu to dou -g ble-c urve-f re ry-- ax ry la pe lim l g u im on to -- as -l o- re ula ry-- rs m irs lato -- axim um-s peed 5 in -l --p g re ula tory prio it-5 urv e-fir t-left g to -- ri re ulato ry-- pri ty-o regulatory--no gu tory m re ula ry-- axim um-s eed-lim -50 n ed fic ath regu st-ri g n o ght wa re ulato ry-- rad rity ver-o reg ulato ry--m axim m-sp d-lim 45 it re g ry re ar- -ro nc h an regu ula reg ulato ry--m ximu -spee en warn rn to --ro vers enfo ad om imit- s ing regu ing reg latory ry-- un ible rc reg gulato --ma imum peed-l in --jun --do g-v --st sto dab -la ed ction war u regu lato eh re latory max it-40 warni -with -a-side ning uble ry op-h p regu latory --sto ere- ou ne t ic s le u -- u m-s ed -lim ng--ju s m warni --double -cu reg latory maxi pe 35 latory nctio -road --tri p on regu latory ple- -sig -red um-s mit- n-with -a-sid -acu te-le ng-- -r rve la na -o u reg latory maxim -- ed-li doub everse -- warnin g--jun e-roa ft -firs regu latory weigh nes-tu ls r-flash u -spe it-30 ction-w d-acu le-tu -c lato --wro t-lim rn-le ing- reg latory-- aximum peed-lim ith-a-s ide-ro te-rig ht warn in rn-fi urve-r t ry-- ng yiel -way it ft-ce lig regu latory--m imum-s ed-limit-25 --ma ht warning--ju ad-per g--ju rs ight t-rig nter war pendic nctio d -la ax nction- ular-le n-with ht ne re gu --m -s pe 0 latory with-a-s imum limit-2 ide-roa d-perp endicul ft ar-right warning-- -a-sid e-road ning regu tory--max um-speed- ximu imit-120 jun ction-w ith-a-s -acu te --do regulatory la regu ory--maxim -speed-l warning-- ide-ro perpen ad- dicula warning- uble regulat y--maximum tor m-spee d-limit-11 0 m-s road-narr -junctio regula --maximu ows r limit-10 warning--ro ad-narrows- n-with regulatory -maximum-speed- -a-sid regulatory- pee left -limit-100 0 warning--road e-road aximum-speed m-speed-limit-led-8 -narrows-right warning--ro regulatory--m regulatory--maximu ad-narrow d-lim warning--curve-left it-led-60 d-limit-led regulatory--maximum-speed-lim warning--curve-right warning--curve regulatory--maximum-spee regulatory--maximum-speed-limit-led-100 it warning--horizontal-alignment-left warning--horizontal-alignment information--tram-bus-stop warning--horizontal-alignment-right rossing-barriers information--trailer-camping ng-with-barriers warning--railroad-c info information--tele phone warning--railroad-crossi -barriers -crossing-without iden road-w rma information- informatio -safety-area com warning--railroad ad-widens warning-- -merge s warning tion informati on-- n--road-b pedestria ns-c ump --traffic informa rossing warning--ro ns-right warning inform tion--pa rking road-wide s-left ation-- turn -first inf inform ormati plem motor warning-- merge ight ing-- ing-road ation on --limi --living-s way --traffic- warn infor treet matio ted-ac warning ffic-m erges-r wind t infor n--in cess- ing-- ne-righ mat road ra ion inform terstate- --t warn dded-la crossing --h at warning ft route info ighway- ion--h urn-le ht enta ht rm ospi int ng --t ft -a ing- icycle n s- e-rig info ation- ersta tal warni --turn-rig oad-first-leight to-th warn --b ildre ds rity- info rm te atio -highw -route ning ch rio s ht rm n- ay-e ng -r st-r war -- sroa ith-p -end raig info at in -g warni rm ion- form as-s inding ning cros s-w ay xit w d-fir war ng-- ssroad highw imals or-g o-st atio -em atio tatio ng-- -roa ry ni n ht info n--d erge n n ding nc --foo d- n- warni ro war ng--c vide ic-a ht-tur icles s-rig rm isa --win d y- --di mest s-rig eh ebri ble ning war ni ning --do -lane ncy-v r-d in in a fa in form tion d-pe cility war fo in fo a -- rs war ing rn du al e erg ck -ro ig wa ning-- em g-ro -in e-r on d s-o a ht rm informarmatio tion--cdead-enons complementary--maximum- at form tion-- n--bu hildre d co -- llin ers rv ti ing ing io atio bike s-sto n m war u ic rn g--fa agg in-c str ross com comp ple c enta en stac pt-b tan s n- co c enta nta en ntary wa --fl irp t-re -c m m ry co com co om ry-- in m om ry ry-- tary -e n--airp-route p wa ing --railr estr ft-o ger e en ple --to ry-- -delin ycle rn ing --ha igh aloo m p ed s-le an dg wa war ing-- road -bum rse g g le ple --o ta o ple ple mple lem rnin --ro oa -cro ro t rn g-- ad -inte ssin ssin nd ort wa ing railr oad ns-c righ rn g--p as r-d ri ple rn e m ry me w-a ass eato g com ntary-- wa rnin g--h ang wa rnin --p the w-b m me --tr n on compl r- k -o uc tary ay-z igh o wa rnin g-- me cti co wa rnin g--o arr ks f m b xce dis wa rnin mpl n -tu ia ple one-d wa rnin g-- com ce rn uck ta c a em wa rnin ning rou wo p -rig s w war ng--sto p-desc d-surfa --tr on t w --sc nda rks m en ht d me war ng--sl hool bout a lem emen p p e war ng--st pery-r one g w le ta ent in info plem oa -r -z e nta ection e ing fo d warni --trail-crosignals -- enta rm rm bo -- warn ng--tex ahea way-tra ing sing ic ffic th-d ati atio tary--c oss rn n-road ry-- r --b ions als wa ee rn ip enta p- -s ire s ts on n us warnin --t-roads ce cks-cr wild-anim warni --traffic ry-- ct --e --e ir e -roads turn --uneve nd nd wa speed-limit complementary--maximum-speed-limit-35 ni g--two- -o -of- k g--tru ry--g ni war hevron complementary- maximum-speed-lim ing ni e ni ng ng f-b m warning--y ep warning-- uil oto warning warnin t-u o r p- way com mplem pleme ar ea co comple com complementary it-15 complementary--max speed-limit-20 complementary--maximum-speed-limit-30 complementary--maximum-speed-limit-25 co mple one-d irectio ple mp co ry-- ne-d me ntary-- plementa aximum-speed-lim com pleme tary--go compl lem enta ection -left nta mentary-- com plemen e comp aximum-spe com en m co tary --turn ight emen ry--maxim mplem hevron lemen complementary--maximum- -m o --maximum nta --tu ry ir ntary tary--c evron-left ry-- rn-r eft tary-- enta imum-speed-limit-40 k igh complementary--m um-speed-l --ke e -l -r ep-r t ch ry--go -spee n ep-l ight ed-lim -right eft d-limit-45 -ri -left imit-55 ght it-75 it-70 Figure A2. MTSD class taxonomy. It contains both novel and known classes. See at full size in [36]. cat t ese+ +ca ox sian lion siam per d skunk r cat otter weasel bobcat pa po cow leo tiger estic gri lar+ cattle zzl gia dom bea y+ nt+ mammal big cat be musteline r pa ra ar nd cc sheep bear wolf oo a pro feline n fox h ua bovid hua cy buffalo chi on i d carnivore a tian dalm antelo shepherd pe collie gulate ruminant canine dog un giraffe german+shepherd dog even-toed ungulate placental mammal rabbit deer elepha nt walr us mo le moose us se a tam ba l pr opo hipp t im ate rodent od pig whale sp d- id to er e ed ors +m gre un h er on blue+ ter gu humpback+ killer+whale at ke av ra lat l y dolp b ms irre e ap be ze os e mous e ha er whale squ ch hin oc im whale in rat pa rh go nz ee r illa Figure A3. AWA2 class taxonomy. It contains both novel and known classes. See at full size in [36]. Sensors 2022, 22, 4389 21 of 22 Shiny_Cowbird Rusty_Blackbird Yellow_headed_Blackbird Brewer_Blackbird Bronzed_Cowbird _Grackle Blue _Jay Flori _Jay Gre Fish rican_ aven Am Laz d_B nting Boat_tailed _Jay da Co ng en Wh g _Cro Crow e unti ntin mm cked_ ak u ite_ sbe Pa igo_B ing ros eak Bu on_ Rave Clar _ey ted ne Gro ros eak w b k Orchard_Oriole Scott_Oriole le d_ uli_ Baltim Towhee ea ros l Western_ inte R b Cardina Bobolin Oriole inch aste Ind Hoode h b Dark_ d_Rosy_Finc unco Blu e_G ch bre h blackbird rple_F jay en _G taile oldfinc ee oldfin _G ore_Orio se_ ed_J cro Eastern_ owh k Pin n d_ e Ro k Meadow rav Ame ean_G d_T G Pu ing ne ey w Wh w_th _Vir _V eo Gre rican_ ble Ev _crow Ye k_N en ite Wa ck_ ade _V eo her ll t lark urop o ea bun N hon atin _W arb r Bla e_h lph _Vir lycatc er Gray en_ utc _Vir ireo rbli hv ary g_W arble r Blu ilad yed d_F atc New World oriole le e rac eo Pr orm ood ing_ War sb E ng pped _Vir o roa eo ireo d r Ph d_e bell Fly c ker r Re llow_ n_ a ot _e ed W ble _ gro e lm Ye ar bler ille _W ar _V e Mo Pa r Ac Le _W ar ble er d W H rn ia ad Ve ast bl r u corvine ar ble ia Ba rmil _Fly finch ie ir _W ar r e as ot Tr rn_ ion_ cat w W rble r llo ee_ Wa rble c e Ba e_ Swa Fly che e S c vireo Y ss ie_ a r h Cli nk_ wa llow atc r e ir W arble r S ll he nn ra a_ ble Br ff_S wa ow Te P nad n_W War ler O o w ll r ld o Ca ins ged _Warb ler _ Sa wn_ allo ow W a g T w oscine in or e Ce e_T hra Sw en_w _whit _Warb rbler ld s d h Bo ar_W rash her sw nd d fly ld a Go ck_a aste lue_W ler ca he mia axw er all Bla re b ated_B rb r tc thr y_ Wa he in n_ Wa g a ow Ba k_thro e_ arble r Pin le_W rbler wa she c xw Bla ing rt a My on_W arbler xw r s W Wil owned_ rbler Wh kingb ing Mo c ite_ ird warbler nge _cr olia _W a ler Am brea Ora Ma g n Warb cky_ Warbler eri st Logg can_ ed_Nu tu n ne erhe Pipit tha tch Ke t_sided_ er ri Sca ad_S rle stnu Warbl sse Brow t_Tana hrike Che ean_ arbler n_C ge C er ul reep r ay_W pa Horne d_La er M r ird Cape _Glos rk Cape_ ed_Warble ing Ovenb Blue_w t Red_win sy_Starlin n_Redstar ge g Spotted_C d_Blackbird America at atb Summer_T ird sted_Ch anager Yellow_brea bler Common_Yellowthroat New World war Gray_Catbird Great_Grey_Shrike Louisiana_Waterthrush Baird_Sparrow Field_Sparrow water thrush Northern_Waterthrush Henslow_Sparrow bird goatsucker Nighthawk Whip_poor_Will Le_Conte_Sparrow humming Chuck_will_Wido w Savannah_Sparrow bird Rufou Ruby_thro s_Hum mingbird Tree_Sparro w king ated_Hum mingbird w ned_Sparro fish Anna_H umm cuc er Green_ ingbird White_crow rrow Fox_Spa ed_Sparrow rrow sparrow wo Green_ Violetear White Ringed_ Kingfisher Clay_ color _tailed _Spa od koo _brea ste Kingfi sh Belte d_Kingfi er Nelso n_Sh arp arrow pe d_Ki Pi ng sher Man ed_Kin fisher n_Sp Lincol _Sparr ow ck grov gf e_C isher Sea side Sparrow er G Ye roov Geoco koo llo Bla w_b e_bille ccyx uc ing_ rrow ow ill Chipp _Spa Sparr Am ck_ ed d_A N bill _C ucko ni er eri se r_ ca Red Dow ort ed_C Hou hoppe ow n_ _b ny hern uck o aquatic T Re Red hre ellie _Wo _F p rr w oo ch ss a d_ _h e_to d_W od lick Gra _S parro co ea pe er at rris ow ed o Pil cka ded _W odpe cker Ha er_S rrow Sparr ow ea de _W ood te d_W oo pe ker c w a Bre g_Sp ated _Spa _ rr c d_ dp cke fly W ood ec r n ro d oo pe ke So e_th ate ow dp ck r du it ro rr r ec er Wh k_th Spa he ke r c Bla per_ ird atc e ck gb lyc we s is Ve yorn l_Kin ed_F _Pe er wren Sa pica r_tail ood d atch r he o tc Tr isso rn_W gbir lyc ca F seabird ly Sc ste Kin d_ m H Pac _Fu Ho ctu _W Wre _F e e G W ay_ sid d ad er orne ific_ lma Bla Soosan_Arn_Gre be W use _W en n Ca rsh na_ ste r e_ w M roli Cre G ga d_P Loo r ck_fo ty_A lbatr be liv all M alb Ca at_ Ro ter Wr en O all r No ns uffin n re r Be ck_W Wre n ar R oo e rth G n d ed atr s ern er n k_ ren a _ Wre pelecanifo gre _ os _b ed H in re wic as d s be te coas d_ tal div _ M auk M er rm seabird er ga ing b ga ns L ird er a ns ote y We er d_ Pie on_G Alb Pig s Ea ed_G be Ho illed_G mot te d_b lba oss atr re rn e Long_tailed_Jaeger cormorant os tro d_G rebe Pomarine_Jaeger s ss larid pelican re uille Frigat auklet re ebird Para ros_Aukle Cres et_Aukle Rhino Lea _Auklet gull Red_faced_ Pelagic_Cormo Brandt_Cormorant Brown_Pe White_P ke ted st_A ce Ca _bille tiwake rnia _Gull tern ukle We s_winged ull Kit Hee ern_G ull Cormorant _G elican _G d_ Her ann_G ll d ull t u lican ge t Ivory ked_Gull leg rant Gull t Least_ _Tern d_ g lifo Re Tern Forsters_Tern Common_Tern Rin _Gull _ u Black_T rn ern uco ring st rm _bac Te Artic_Tern ian Gla Elegant_ Slaty Casp Figure A4. CUB class taxonomy. It contains both novel and known classes. See at full size in [36]. References 1. Masana, M.; Ruiz, I.; Serrat, J.; van de Weijer, J.; Lopez, A.M. Metric Learning for Novelty and Anomaly Detection. In Proceedings of the British Machine Vision Conference (BMVC), Newcastle, UK, 3–6 September 2018. 2. Lee, K.; Lee, K.; Lee, H.; Shin, J. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Montreal, QC, Canada, 3–8 December 2018. 3. Lee, K.; Lee, H.; Lee, K.; Shin, J. Training confidence-calibrated classifiers for detecting out-of-distribution samples. In Proceedings of the International Conference on Learning Representations (ICLR), Vancouver, BC, Canada, 30 April–3 May 2018. 4. Hendrycks, D.; Gimpel, K. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In Proceedings of the International Conference on Learning Representations (ICLR), Toulon, France, 24–26 April 2017. 5. Lee, K.; Lee, K.; Min, K.; Zhang, Y.; Shin, J.; Lee, H. Hierarchical Novelty Detection for Visual Object Recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA, 18–22 June 2018. 6. Boult, T.E.; Cruz, S.; Dhamija, A.R.; Gunther, M.; Henrydoss, J.; Scheirer, W.J. Learning and the unknown: Surveying steps toward open world recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA, 27 January–1 February 2019. 7. Denzler, J.; Rodner, E.; Bodesheim, P.; Freytag, A. Beyond the closed-world assumption: The importance of novelty detection and open set recognition. In Proceedings of the German Conference on Pattern Recognition (GCPR) (Unsolved Problems in Pattern Recognition Workshop (UPPR)), Saarbrücken, Germany, 3 September 2013. 8. Wang, F.; Xiang, X.; Cheng, J.; Yuille, A.L. Normface: L2 hypersphere embedding for face verification. In Proceedings of the 25th ACM International Conference on Multimedia, Mountain View, CA, USA, 23–27 October 2017. 9. Zhou, D.W.; Yang, Y.; Zhan, D.C. Learning to classify with incremental new class. IEEE Trans. Neural Netw. Learn. Syst. 2021, 33, 2429–2433. [CrossRef] [PubMed] 10. Wali, S.B.; Abdullah, M.A.; Hannan, M.A.; Hussain, A.; Samad, S.A.; Ker, P.J.; Mansor, M.B. Vision-Based Traffic Sign Detection and Recognition Systems: Current Trends and Challenges. Sensors 2019, 19, 2093. [CrossRef] [PubMed] Sensors 2022, 22, 4389 22 of 22 11. Chen, J.; Jia, K.; Chen, W.; Lv, Z.; Zhang, R. A real-time and high-precision method for small traffic-signs recognition. Neural. Comput. Appl. 2022, 34, 2233–2245. [CrossRef] 12. Zhu, Y.; Yan, W.Q. Traffic sign recognition based on deep learning. Multimed. Tools. Appl. 2022, 81, 17779–17791. [CrossRef] 13. Dewi, C.; Chen, R.C.; Jiang, X.; Yu, H. Deep convolutional neural network for enhancing traffic sign recognition developed on Yolo V4. Multimed. Tools. Appl. 2022, 1–25. [CrossRef] 14. Pang, G.; Shen, C.; Cao, L.; Hengel, A.V.D. Deep Learning for Anomaly Detection: A Review. ACM Comput. Surv. 2021, 54, 1–38. [CrossRef] 15. Fort, S.; Ren, J.; Lakshminarayanan, B. Exploring the limits of out-of-distribution detection. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Virtual, 6–14 December 2021. 16. Xian, Y.; Lampert, C.H.; Schiele, B.; Akata, Z. Zero-shot learning—A comprehensive evaluation of the good, the bad and the ugly. IEEE Trans. Pattern Anal. Mach. Intell. 2018, 41, 2251–2265. [CrossRef] 17. Xian, Y.; Lorenz, T.; Schiele, B.; Akata, Z. Feature generating networks for zero-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA, 18–22 June 2018. 18. Seo, S.; Kim, J. Hierarchical Semantic Loss and Confidence Estimator for Visual-Semantic Embedding-Based Zero-Shot Learning. Appl. Sci. 2019, 9, 3133. [CrossRef] 19. Geng, C.; Huang, S.j.; Chen, S. Recent advances in open set recognition: A survey. IEEE Trans. Pattern Anal. Mach. Intell. 2020, 43, 3614–3631. [CrossRef] [PubMed] 20. Bertinetto, L.; Mueller, R.; Tertikas, K.; Samangooei, S.; Lord, N.A. Making better mistakes: Leveraging class hierarchies with deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Virtual, 13–19 June 2020. 21. Garnot, V.S.F.; Landrieu, L. Leveraging Class Hierarchies with Metric-Guided Prototype Learning. In Proceedings of the British Machine Vision Conference (BMVC), Virtual, 22–25 November 2021. 22. Brust, C.A.; Barz, B.; Denzler, J. Making every label count: Handling semantic imprecision by integrating domain knowledge. In Proceedings of the International Conference on Pattern Recognition (ICPR), Milan, Italy, 10–15 January 2021. 23. Brust, C.A.; Denzler, J. Integrating domain knowledge: Using hierarchies to improve deep classifiers. In Proceedings of the Asian Conference on Pattern Recognition (ACPR), Auckland, New Zealand, 26–29 November 2019. 24. Buzzelli, M.; Segantin, L. Revisiting the CompCars Dataset for Hierarchical Car Classification: New Annotations, Experiments, and Results. Sensors 2021, 21, 596. [CrossRef] [PubMed] 25. Wu, T.Y.; Morgado, P.; Wang, P.; Ho, C.H.; Vasconcelos, N. Solving long-tailed recognition with deep realistic taxonomic classifier. In Proceedings of the European Conference on Computer Vision (ECCV), Glasgow, UK, 23–28 August 2020. 26. Chen, Q.; Liu, Q.; Lin, E. A knowledge-guide hierarchical learning method for long-tailed image classification. Neurocomputing 2021, 459, 408–418. [CrossRef] 27. Liu, W.; Wen, Y.; Yu, Z.; Yang, M. Large-margin softmax loss for convolutional neural networks. In Proceedings of the International Conference on Machine Learning (ICML), New York, NY, USA, 19–24 June 2016. 28. Liu, W.; Wen, Y.; Yu, Z.; Li, M.; Raj, B.; Song, L. Sphereface: Deep hypersphere embedding for face recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 21–26 July 2017. 29. Wang, H.; Wang, Y.; Zhou, Z.; Ji, X.; Gong, D.; Zhou, J.; Li, Z.; Liu, W. Cosface: Large margin cosine loss for deep face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA, 18–22 June 2018. 30. Deng, J.; Guo, J.; Xue, N.; Zafeiriou, S. Arcface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 16–20 June 2019. 31. Zhang, X.; Zhao, R.; Qiao, Y.; Wang, X.; Li, H. Adacos: Adaptively scaling cosine logits for effectively learning deep face representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 16–20 June 2019. 32. Zhao, K.; Xu, J.; Cheng, M.M. RegularFace: Deep Face Recognition via Exclusive Regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 16–20 June 2019. 33. Welinder, P.; Branson, S.; Mita, T.; Wah, C.; Schroff, F.; Belongie, S.; Perona, P. Caltech-UCSD Birds 200; Technical Report CNS-TR-201; Caltech: Pasadena, CA, USA, 2010. 34. Zhu, Z.; Liang, D.; Zhang, S.; Huang, X.; Li, B.; Hu, S. Traffic-sign detection and classification in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 26 June–1 July 2016. 35. Ertler, C.; Mislej, J.; Ollmann, T.; Porzi, L.; Kuang, Y. Traffic Sign Detection and Classification around the World. In Proceedings of the European Conference on Computer Vision (ECCV), Glasgow, UK, 23–28 August 2020. 36. GitHub Repository. Available online: https://github.com/idoiaruiz/HierarchicalCosineLoss.git (accessed on 21 April 2022). 37. Deng, J.; Berg, A.C.; Li, K.; Li, F.-F. What does classifying more than 10,000 image categories tell us? In Proceedings of the European Conference on Computer Vision (ECCV), Hersonissos, Crete, Greece, 5–11 September 2010.