Neural Computing and Applications https://doi.org/10.1007/s00521-022-07031-9 (0123456789().,-volV)(0123456789(). ,- volV) ORIGINAL ARTICLE Don’t miss the mismatch: investigating the objective function mismatch for unsupervised representation learning Bonifaz Stuhr1,2 • Jürgen Brauer2 Received: 27 May 2021 / Accepted: 30 January 2022 Ó The Author(s) 2022 Abstract Finding general evaluation metrics for unsupervised representation learning techniques is a challenging open research question, which recently has become more and more necessary due to the increasing interest in unsupervised methods. Even though these methods promise beneficial representation characteristics, most approaches currently suffer from the objective function mismatch. This mismatch states that the performance on a desired target task can decrease when the unsupervised pretext task is learned too long–especially when both tasks are ill-posed. In this work, we build upon the widely used linear evaluation protocol and define new general evaluation metrics to quantitatively capture the objective function mismatch and the more generic metrics mismatch. We discuss the usability and stability of our protocols on a variety of pretext and target tasks and study mismatches in a wide range of experiments. Thereby we disclose dependencies of the objective function mismatch across several pretext and target tasks with respect to the pretext model’s representation size, target model complexity, pretext and target augmentations as well as pretext and target task types. In our experiments, we find that the objective function mismatch reduces performance by  0.1–5.0% for Cifar10, Cifar100 and PCam in many setups, and up to  25–59% in extreme cases for the 3dshapes dataset. Keywords Objective function mismatch  Metrics mismatch  Unsupervised  Self-supervised  Representation learning  Pattern recognition 1 Introduction unsupervised learning methodologies only a few of them can be used across the wide spectrum of promising Unsupervised Representation Learning is a promising approaches. This is one reason why the linear evaluation approach to learn useful features from huge amounts of protocol is now commonly used [9, 14, 16, 21, 32, 37, 46], data without human annotation effort. Thereby, a common which trains a linear model for a target task on-top of the evaluation pattern is to train an unsupervised pretext model representations of an unsupervised pretext model. In this on different datasets and then test its performance on work, we show that simply training a target model for several target tasks. Because of the huge variety of target different layers of the pretext model does not yield the tasks and preferred representation characteristics, the entire picture of the training process and leads to a loss of evaluation of these methods is challenging. In recent work, useful temporal information about learning. It is already a large number of evaluation metrics have been proposed known in literature, that succeeding in a pretext task can be [24, 39, 40, 45], but because of the fast changes in the reason why the model fails on the target task. Here we propose that the linear evaluation protocol does not capture this properly. Therefore, we extend this protocol and & Bonifaz Stuhr bonifaz.stuhr@hs-kempten.de address the question of when succeeding in a pretext task hurts performance and how much. We train target models Jürgen Brauer juergen.brauer@hs-kempten.de on representations obtained from different training steps or epochs of the pretext model and plot target and pretext 1 Department of Computer Science, Autonomous University of model metrics in comparison, as shown in Fig. 1. Thereby Barcelona, 08193 Bellaterra, Barcelona, Spain we observe that training an unsupervised pretext model too 2 Department of Computer Science, University of Applied long can lead to an objective function mismatch [41, 58] Sciences Kempten, 87435 Kempten, Germany 123 Neural Computing and Applications 2 Related work 2.1 Unsupervised representation learning Many unsupervised representation learning algorithms are based on self-supervised learning [26, 52, 53], which obtains labels directly from data without human annotation to define a pretext task. There are several approaches to self-supervision: Generation-based self-supervison examines the generation Fig. 1 (Bottom left) Evaluation loss of a pretext autoencoder trained of an arbitrary output from a learned representation of the on Cifar10. (Top left) Best evaluation losses of linear target models given input. One line of work improves on autoencoders trained for classification on the representations of the pretext [51] and variational autoencoders [31] by defining gener- autoencoder from different pretext training epochs. (Right) Evalua- tion loss curves from each linear target model. Colors correspond to ation-based pretext tasks which lead to representations the current epoch of pretext task training, and each value is obtained valuable for required target tasks (e.g., object classification by 5-fold cross-validation. An objective function mismatch occurs or detection). Examples are denoising [8, 61], colorization around pretext training epoch 40, from which the target loss increases [34, 35, 73], or inpainting of images [47, 67]. Recently, a second line of work based on GANs [17] emerged, which between the objectives used to train both models. This adjusts their latent space for representation learning, for mismatch leads to a drop in performance on the target task, example by constraining [50] or changing [14] the archi- while the pretext model and the target models still con- tecture. In a third line of research, an autoregressive, verge correctly, which can be seen in Fig. 1. To quantify transformer-based model achieved state-of-the-art perfor- our results, we define soft and hard versions for two simple mance on visual representation learning by sequential and general evaluation metrics - the metrics mismatch and image generation [20]. Generation-based self-supervison is the objective function mismatch - formally. With these applied to other modalities as well, e.g., audio [20] or video metrics, we then evaluate different image-based pretext [57, 62]. task types for self-supervised learning by using the linear evaluation protocol. Context-based self-supervison recently has moved more Our contributions can be summarized as follows: and more away from autoencoding data: Early approaches utilize spatial context structure by defining pretext tasks for – We propose hard and soft versions of general metrics to context generation, like image inpainting [47] or denoising, measure and compare mismatches of (unsupervised) as a weak form of inpainting [8, 61]. In contrast, approa- representation learning methods across different target ches for context prediction do not create any image and, for tasks (Sect. 3 and 4). To the best of our knowledge, this example, try to leverage the knowledge obtained by pre- has not been done before. dicting patch positions [12, 43]. Spatial context can also be – We discuss the usability and stability of our protocols encoded by predicting transformations, which has led to a on a variety of pretext and target tasks (Sect. 6.2). line of research focusing on autoencoding transformations – In our experiments we qualitatively show dependencies rather than data [16, 37, 49, 72]. Recently, the context- of the objective function mismatch with respect to the based similarity approach of contrastive learning [19], pretext model’s representation size (Sect. 6.3), target which utilizes context information between negative and model complexity (Sect. 6.4), pretext and target aug- positive pairs, gained popularity and achieved promising mentations (Sect. 6.5) as well as pretext and target task results [9, 11, 21, 36, 56]. Contrastive Learning has been types (Sect. 6.6). linked to mutual information maximization [38, 68], which – We find that the objective function mismatch can in ongoing work is used to define pretext tasks through reduce performance on various benchmarks. Specifi- context-based similarity as well [4, 24]. Context similarity cally, we observe a performance decrease by  0.1– by pseudo-labeling through clustering methods is another 5.0% for Cifar10, Cifar100 and PCam, and up to line of research [7, 71]. Self-supervised relational reason-  25–59% in extreme cases for the 3dshapes dataset ing combines context-based similarity and context-based (Sect. 6). structure by discriminating how entities relate to them- selves and to other entities and has also been linked to mutual information maximization [46]. Context-based 123 Neural Computing and Applications approaches are applied to other data modalities as well, interesting dependencies of the objective function e.g., point clouds [69] or video [15, 28]. mismatch. Other unsupervised representation learning methods for example combine multiple self-supervised approaches [10, 29, 48, 63], use meta learning [25, 27, 41, 54] or 3 Hard metrics mismatch metric learning [6, 65] to learn unsupervised learning rules, or rely on self-organization [1, 58]. With the objective function mismatch, we want to measure the mismatch of two objectives while training a model on a 2.2 Analyzing unsupervised representation (unsupervised) pretext task and using its representations to learning train another model on a target task. In general, we can measure the mismatch of two comparable metrics, if one Changing the underlying model is one common theme to metric is captured during training of a single pretext model compare different unsupervised learning techniques [18]. and the other is captured for each target model fully trained Here, a well known finding is that a larger representation on the representations of different steps or epochs of the size significantly and consistently increases the quality of pretext model. Two comparable metrics, for example, are the learned visual representations [32]. classification accuracies for the pretext and target task, because they use the same measurement unit and scale. As Varying the amount of data samples has led to interesting illustrated in Fig. 2, the metric values of the target models observations as well [42, 59]. For example in [3], it is form a curve over the course of learning. Between a metric shown that unsupervised learning is capable to learn fea- value on this curve and the corresponding metric value on tures of early layers from a single image. the pretext model curve, we can define the metrics mis- Analyzing self-supervised learning across target domains match (M3) for a certain step (or epoch) in training by is another way to define and evaluate benchmarks for calculating their distance. unsupervised approaches [8, 42, 64]. Zhai et al. [70] define More formally, let M P ¼ ðmP1 ; :::; mPn Þ denote an n-tuple good representations as those that adapt to diverse, unseen of values from a metric used to measure pretext model P tasks with few examples. for different steps S ¼ ðs1 ; :::; sn Þ. The length n of the tuple Furthermore, there exist works where the underlying is usually given by a convergence criterion C on the metric model, the amount of data samples and the target domain is of model P during training. Furthermore, let M T ¼ analyzed collectively [18, 44]. ðmT1 ; :::; mTn Þ denote an n-tuple of values from a comparable Other investigations of unsupervised learning focus on the metric used to measure target model T. M T is of the same effect of the multitask pretext learning [13, 55], evaluate length and order as M P and all values are calculated at the the disentanglement of representations [39], investigate the same training steps S of M P . Thereby the target model T is positive effects of unsupervised learning regarding robustness [23], or provide a theoretical analysis of con- trastive learning [2, 66]. The objective function mismatch in unsupervised learning is not unknown. Some works directly or indirectly observed that learning a pretext task too long may hurt target task performance, but made no further investigations on this topic [32, 39, 64]. Other works sometimes showed performances of linear target models over training epochs, but did not examine or define the objective function mis- match in detail [58, 70]. Instead, unsupervised multi-task learning and meta learning are proposed as approaches to Fig. 2 (Left) Intuition behind MM3: In this case, both metrics lower the objective function mismatch [13, 41]. In contrast, measure a classification error in %. The pretext metric (solid curve) is measured on the pretext task of predicting rotations with a ResNet18 this work focuses on defining general protocols to measure model and the target metric (dotted curve) on the fully trained Cifar10 mismatches of metrics over the course of pretext task classification task. When divided by the number of measurements, the training when a target task is trained on top of the pretext discrete area enclosed by the target and pretext task curves model’s representations. To the best of our knowledge, this corresponds to the MM3 of the entire training process. (Right) Metrics Mismatch M3 plotted during training: We observe a common has not been done before. Furthermore, we highlight behavior where the mismatch increases as training progresses. important properties of our evaluation protocols and Additionally, we show the stability (þ; ) of M3 and MM3 across a 5-fold cross-validation 123 Neural Computing and Applications fully retrained for every step si in S on the representations Fig. 2, analogous to the way a loss is measured and plotted of model P at this step before we measure mTi . during training. Definition 1 The hard Metrics MisMatch (M3) between 3.1 Hard objective function mismatch mTi and mPi at step si is defined as: M3ðmTi ; mPi Þ:¼mTi  mPi ð1Þ Naively, we could compare the objective functions of the target and pretext task by using M3, which we define as the where mTi and mPi are single values measured with com- hard objective function mismatch. In most cases, however, parable metrics at step si .1 the objective functions used to train the pretext model and If M3 [ 0 the performance of the target model is lower the target models are not directly comparable. This is due than the performance of the pretext model at step si . In to the usage of different objective functions for both model contrast, M3  0 represents the desired case in unsuper- types, which, i.e., use different (non)linearities. But for vised representation learning, where target model perfor- some pretext tasks simple, comparable metrics can be mance is the same or above the pretext model performance defined. These metrics can be used as a proxy to measure at step si . In our case, we measure mTi and mPi over the the objective function mismatch in a general and compa- entire evaluation dataset for every step si in S. We plot rable manner. A well known example is the accuracy M3ðmTi ; mPi Þ for the pretext task of predicting rotations and metric, which can be used on the self-supervised tasks of the target task of Cifar10 classification during training in predicting rotations [16] and the state-of-the-art approach Fig. 2. This shows that our metric captures the behavior of of contrastive learning [9]. But comparable metric pairs the target task performance regarding the pretext task can not always be found easily. For example, if we train a performance, and we observe an increasing mismatch as variational autoencoder and later use its representation for training progresses. To capture the mismatch of the entire a classification target task, it does not make sense to define training procedure with respect to the target task in a single a pixel-wise error between the given and generated images value, we can now define the mean hard metrics mismatch as a comparable pretext task metric. To achieve a compa- (MM3) as the mean bias error between M T and M P . rable measurement for this situation, and on the loss curves in general, one could think of individual normalization Definition 2 The Mean hard Metrics MisMatch (MM3) techniques between objective function pairs. However, we between M T and M P is defined as: want to be practical and define a measure, which can be 1 X used independently of the objective function pairs for every MM3ðM T ; M P Þ:¼ ðmT  mPi Þ ð2Þ pretext and target model combination. Furthermore, in n 0\i  n i practice, we might be especially interested in how much where M T and M P are tuples measured with comparable the target task mismatches with the pretext task if a mis- metrics until the pretext model converges at step sn . match decreases target performance. This is why we define soft versions of our measurements. MM3 measures the bias of the target model metric to the pretext model metric. For positive or negative values of MM3, we can make similar observations as for M3, but 4 Soft metrics mismatch they now account for the tendency of the entire training process and not for a single step si . In general, the mean To bypass objective function pair normalization, we define bias error can convey useful information, but it should be the soft metrics mismatch (SM3) directly on the target interpreted cautiously because there are special cases metric. Thereby, we no longer take the exact improvement where positive and negative values cancel each other out. of the pretext metric into account, we only care about its In our case, this can happen, for example, when learning convergence. Since we now have no exact information the pretext task is very useful for the target task early in about the pretext metric curve, we define SM3 for the training but hurts the target performance equally strong current step si between the current target metric value and later on when the pretext task is sufficiently solved. We the previously or currently occurred minimal target metric simply capture this behavior by measuring and plotting M3 value: individually for the metric values of each step si as in Definition 3 The Soft Metrics Mismatch (SM3) between M T and M P at step si is defined as: 1 Note that we define our measurements only for the case where SM3ðmTi Þ:¼mTi  min ðmTj Þ ð3Þ lower metric values correspond to better performance. The definition 0\j  i for the opposite case arises naturally by changing maximum and minimum operations and/or subtraction orders. 123 Neural Computing and Applications   where min0\j  i ðmTj Þ is the previously or currently occur- OFMðmTi Þ:¼SM3 NðmTi Þ ð7Þ red minimal target metric value. 8 > 100  x > < T mT1 gt; x  mTb SM3 has a slightly different meaning compared to M3: m1  mTb NðxÞ :¼ It equals zero if mTi is a minimal metric value and is pos- > > 0 mT1 ¼ mTb ¼ x : itive if mTi is higher than the previously occurred minimal 1ðspecialcaseÞ mT1 ¼ mTb lt; x metric value. We want to point out that the only way we ð8Þ incorporate the pretext metrics into this measurement is by where mT1 is the loss value of the target model trained on an making sure that the pretext model does not overfit and has untrained pretext model (s1 ¼ 0) and b¼ not yet converged. Again, we measure mTi and mPi over the T argmin0\i  n ðmi Þ denotes the index of the minimal target entire evaluation dataset for every step si in S and plot T SM3ðmTi Þ. A common case is shown in Fig. 3, which loss value. We then use M^ ¼ ðNðmT1 Þ; :::; NðmTn ÞÞ to cal- captures the behavior of pretext model training with respect culate the OFM. to the target model. Here we observe zero soft mismatch early in training followed by increasing soft mismatch until The intuition behind this normalization is that we pretext model convergence. Again, we can capture the declare mTb as the value where the pretext model has mismatch of the pretext task with respect to the target task learned all of the target objective, it was able to learn (with for the entire training process until pretext model conver- this setting) and mT1 as the value where the model has gence as the mean bias error of every metric value mTi and learned nothing of the target objective. Now we measure its minimal metric value: with OFMðmTi Þ for what percentage the learning of a pre- Definition 4 The Mean Soft Metrics Mismatch (MSM3) text objective hurts the maximum achieved target perfor- between M T and M P is defined as: mance at step si . An example is illustrated in Fig. 3.   Furthermore, we can normalize the other soft measure- T 1 X T T MSM3ðM Þ:¼ m  min ðmj Þ ð4Þ ments from Eqn. 5 and 6 analogs to Eq. 7. n 0\i  n i 0\j  i The OFM is a general measure, which can be used for when the pretext model convergences at step sn . pretext and target models where no good proxy metrics can be defined. With the OFM we are able to compare mis- MSM3 can either be zero, if no mismatch occurs, or matches across different pretext and target task objectives positive, if there is a mismatch. Therefore, using MSM3 and their combinations. We propose these measurements to brings the benefit that positive values can not be canceled obtain quantitative and therefore comparable results for out by negative values. Furthermore, we define the maxi- individual pretext tasks. To get the best information about mum occurring mismatch mSM3 and the mismatch at the the training process, we encourage to plot the curves pretext model convergence cSM3. We are especially formed by our metrics as well. We want to point out that interested in cSM3, since it measures the representations our metrics are not intended to measure target task per- one would naively take for the target task: formance, they measure how much the performance on a cSM3ðM T Þ :¼ SM3ðmTn Þ ð5Þ target task can decrease when an (ill-posed) pretext task is   learned too long. Now, to understand the OFM further, we mSM3ðM T Þ :¼ max SM3ðmTi Þ ð6Þ take a look at some cases: 0\i  n OFMðmTi Þ ¼ 0: In this case, solving the pretext task 4.1 Soft objective function mismatch objective did not hurt the performance of the target task objective at this point in training. Now we can use SM3 to measure a soft form of the OFMðmTi Þ ¼ x: Solving the pretext objective did hurt objective function mismatch on the loss curve obtained by the performance of the target objective at this point in the target models. However, the values of these measure- training by x% of what the model has learned. Therefore, ments lie in a range, which depends on the target objective we should have stopped training earlier. It is not guaran- function. Therefore, they are not directly comparable to the teed that longer training would hurt performance even measurements on loss curves from other target tasks. This more, but a growing OFM curve or MOFM is a good is why we normalize the measurements of the target metric indicator for that. to percentage range and define the objective functions OFMðmTi Þ [ 100 The target objective performance is mismatch (OFM) as follows: worse than for the untrained model at this point in training. Definition 5 The Soft Objective Function Mismatch MOFMðM T Þ ¼ 1 Solving the pretext objective hurts (OFM) between M T and M P at step si is defined as: the performance of the target objective from the point of 123 Neural Computing and Applications learning a nonlinear head, as suggested in [9]. We show that mismatches account for other architectures as well, by carrying out additional evaluations using ResNets [22] in Table 3 and Appendix C. Target tasks We evaluate our metrics on image-based tar- get tasks. For coarse-grained classification, we use Cifar10, Cifar100 [33] and the coarse-grained labels of 3dshapes [5]. For fine-grained classification, we use the PCam dataset [60] and the fine-grained labels of 3dshapes. Target models Following the linear evaluation protocol, we Fig. 3 We measure the OFM instead of SM3 by normalizing the metric values with Equation 8. For visualization, we additionally shift use a single, linear dense layer (FC) as a target model with the normalized metric values such that they lie in [0, 100] by a softmax activation. To evaluate our metrics for other subtracting the minimal measurement. (Left) Intuition for the MOFM: target models, we use a two MLP (2FC) and a three-layer When divided by the number of measurements, the discrete area MLP (3FC). enclosed by the target metric values and their previous minimal target metric values correspond to the OFM of the entire training process. Augmentations We make sure not to compare augmenta- The red arrow shows the cOFM and the black arrow is the mOFM. In tions instead of pretext tasks by following Chen et al. [9] this case, the target metric measures the cross-entropy loss of each fully trained target model on a Cifar10 validation set. (Right) The for our base augmentations to which we add the pretext OFM plotted during training: We observe an increasing mismatch task-specific augmentations for pretext task training and starting around epoch 600. Additionally, we show the range (þ; ) of evaluation. For the target task, we use the base training and the OFM and MOFM across a 5-fold cross-validation evaluation augmentations of Chen et al. [9]. initialization. Because we have learned essentially 0% Optimization Our models are trained using the Adam about the target objective in the training process, there is no optimizer [30] with standard parameters and batch size interval to be used for normalization. Therefore, we inter- 2048 without any regularization instead of batch normal- pret this case as if the model has an infinite mismatch as ization. For our ResNets, we additionally use a weight soon as the model forgets something about the target decay of 1e4. objective. Mismatch evaluation All reported values are determined by 5-fold cross-validation. We use standard early stopping (from tf.keras) as convergence criterion on the pretext 5 Experimental setup evaluation curve with a minimum delta (threshold) of 0 and patience of 3. We change the patience in some experiments In our experiments, we focus on image-based self-super- of Tables 1 and 3 to get a reasonable convergence epoch. vised learning. However, it is likely that other target For more details, we refer to Appendix B. When calcu- domains show mismatches as well, e.g., [13]. lating our metrics, we estimate target values of missing Pretext tasks For generation-based self-supervision, we epochs with linear interpolation to save computation time. evaluate the approaches of autoencoding data from In our case, SM3 and MM3 are measured on the target task autoencoders (CAE) and color restoration (CCAE) as accuracy. suggested by Chen et al. [9]. To evaluate context structure Implementation Our implementation is available at https:// generation we use denoising autoencoders (DCAE). Spatial github.com/BonifazStuhr/OFM. context structure is evaluated via autoencoding transfor- mations by predicting rotations [16] (RCAE). For context- based similarity methods, we follow the state-of-the-art 6 Evaluation contrastive learning approach from Chen et al. [9] (SCLCAE). We refer to the literature for first glances into In the following, we show results of most pretext and target mismatches for VAEs, meta-learning [41] and self-orga- tasks we have evaluated. We refer to Appendix C for nization [58]. additional, more detailed evidence. Since we capture our Pretext models Unless stated otherwise, we use a four-layer metrics during training, all mismatches are measured on CNN as encoder. For the autoencoding data approaches, we the evaluation dataset. use a four-layer decoder with transpose convolutions, for rotation prediction a single dense layer and for contrastive 123 Table 1 MOFM, cSM3 and MM3 of the models from Fig. 4 Neural Computing and Applications CAE (Cifar10) DCAE (Cifar10) CCAE (Cifar100) CCAE (PCam) RCAE (PCam) SCLCAE (3dshapes) cSM3 MOFM cSM3 MOFM cSM3 MOFM cSM3 MOFM cSM3 MOFM MM3 cSM3 MOFM MM3 Rep. size 2x2x4 0.00 0.00 0.05 0.00 0.28 1.54 4.98 9.28 5.38 4.15 22.26 26.34 1 7.99 2x2x32 0.07 1.99 0.00 3.20 0.65 3.64 5.17 34.30 3.34 7.92 21.09 12.98 36.39 57.67 2x2x128 0.20 10.10 0.06 5.51 0.51 0.81 0.32 0.10 1.03 4.04 23.47 8.14 22.65  66.19 2x2x256 0.75 11.14 0.69 5.17 0.17 0.00 0.43 0.87 0.44 0.00 27.60 6.52 27.65 65.77 2x2x512 0.43 5.28 0.36 1.25 0.00 0.00 0.20 0.07 0.18 0.00  28.03 5.96 27.78 63.61 2x2x1024 0.24 0.25 0.03 0.00 0.00 0.00 0.00 0.00 0.09 0.00 26.56 4.76 32.70 57.92 Target model FC 0.75 11.14 0.69 5.17 0.00 0.00 0.00 0.00 1.03 4.04 23.47 6.52 27.65 65.77 2FC 0.03 5.68 0.00 5.14 0.08 0.00 0.00 0.00 0.31 0.76 28.56 1.84 103.30 70.49 3FC 0.03 3.94 0.00 3.17 0.12 0.02 0.08 0.00 0.37 0.61  29.61 0.91 258.18  71.26 Augmentations All 0.75 11.14 0.69 5.17 0.17 0.00 0.43 0.87 1.03 4.04 23.47 6.52 27.65  65.77 NoJitter 0.99 10.99 0.50 2.33 – – – – 0.58 0.10  28.60 0.00 0.01 37.92 NoJitterNoFlip 1.00 12.51 0.55 1.73  – – – 1.51 7.33 10.60 0.00 0.00 35.95 NoFlip     0.20 0.00 0.41 0.04 - – – – – – The smallest mismatches for each setup are printed in bold MM3 is measured on the target and pretext task classification error. cSM3 is measured on the target task classification error and corresponds to the accuracy we would lose when we naively train the target model after pretext model convergence. Values are obtained by 5-fold cross-validation. We show the stability of each measurement in Tables 5, 6 and 7 of Appendix B 123 Neural Computing and Applications Fig. 4 (Top) Impact of different pretext model representation sizes on the OFM. (Middle) OFM for the linear target model and nonlinear target models trained on our pretext model. (Bottom) OFM for the linear target model and for the pretext models trained on fewer augmentations. First, we removed the color jitter and then the vertical flip from the augmentations. The target models of SCLCAE were trained on 3dshapes to predict the object hue 6.1 Mismatch and convergence fluctuations with pretext fluctuations. In general, we observe that as long as we calculate the OFM across a fair For our measurements, we make sure to use metric value amount of cross-validations (in our case 5), we can make pairs from models that do not overfit. We achieve this by statements about the mismatch. We measure our metrics on applying a convergence criterion on the pretext task and by the mean losses during 5-fold cross-validation instead of using the best metric values from each target model eval- calculating them five times and taking the average. For M3 uation curve. As shown in Appendix C, most observations both variants are equivalent and for the OFM measuring on in our experiments are independent of the use of a con- the mean losses leads to a lower bound in the case where vergence criterion, if pretext models are trained long all models converge at step sn (see Appendix A for the enough and without overfitting. Furthermore, we observe a simple proofs). We prefer to measure our metrics on the common behavior in Fig. 1: Target models trained on mean losses, since this avoids mismatches occurring just in higher epochs of the pretext model tend to converge faster. some validation cycles due to small fluctuations of the This indicates that longer training of the pretext task tends underlying training procedure. An example is shown in to create easier separable representations which may translucent red in Fig. 3 at the beginning of training. We mismatch with the class label. want to point out that the training and validation data differ slightly in every round because of the cross-validation 6.2 Stability setup. This increases the instability, but shows the general behavior of the metrics for the underlying data distribution. To evaluate the stability of our measurements, we show the In Appendix C we compare the instability of partially mismatches of the entire training process and their range measured mismatches using linear interpolation with mis- (þ; ) using 5-fold cross-validation in Figs. 2 and 3. The matches measured for every pretext training epoch and range of all other models, we have trained is shown in observe a similar instability. However, when using the Appendix C. We observe that M3 generally seems more OFM in practice to compare models on a finer scale, we stable than SM3 or the OFM, since it does not rely so recommend to search for the actual minimal target metric heavily on the target metric values, which can be quite value, since the OFM relies on this value at each step. unstable. The instability of the target task mismatch is However, when tuning a model for maximum performance, captured in M3, but does not matter that much in the one searches for this value. Thereby looking at the OFM overall measurement for most cases. This is favourable if a curve gives good indications in which interval one should stable value is desired and unfavourable if one wants to search. This makes this protocol useful for performance capture the instability of the target task training process tuning, if enough computational power is available. explicitly. Furthermore, M3 is able to compensate target 123 Neural Computing and Applications 6.3 Dependence on representation size which do not fit properly with the target task. Again we found an exception where the MOFM is lower for linear We hypothesize that large representation sizes tend to models when predicting the object hue after contrastive lower the OFM, which could be one reason why repre- learning which can be appointed to the color-selective, sentation sizes are large in unsupervised learning. To affirm random features of the untrained pretext model. this hypothesis empirically, we train our pretext models with varying representation sizes on different target tasks 6.5 Dependence on augmentations while fixing all other model parameters. Figure 4 and Table 1 show that the OFM tends to decrease when we We vary the augmentations used for the pretext and target enlarge the representation size. A reason for that might be model by removing the color jitter and the image flip from that target models can exploit the high dimensional space our base augmentations successively. Figure 4 shows that of large representations to find better fitting clusters for augmentations can have a positive or negative impact on their target task. We found an exception of this behavior, the mismatch. E.g., when predicting the object hue, the ill- where we use larger representations of SCLCAE for the posed color jitter augmentation increases the mismatch easy task of object hue prediction. Here, the target models significantly. trained on the untrained pretext models with larger repre- sentation sizes already achieve high performance due to a 6.6 Dependence on target task type larger number of color-selective, random features. Further learning of the pretext model, in this case, does not lead to Here we use our metrics to examine findings stated in a high-performance gain and forgetting these sensitive [13, 32] and [64, 70], where it is argued that some pretext random features during training leads to a high mismatch. tasks are better suited for different target tasks. We fix the Additionally, we observe that mismatches decrease, when underlying data distribution by using the 3dshapes dataset we decrease the representation size for generation-based and train our target models for the different tasks. These methods. A reason could be that the pretext models are tasks require a generic understanding of the scene like forced to generalize to solve the target task for small rep- coarse-grained knowledge about object type and hue and resentation sizes due to the limited amount of features in fine-grained knowledge about shapes, positions and scales. the bottleneck, or simply underfit on the pretext task. In Fig. 5 and Table 2, we observe that pretext models tend to learn pretext task specific features and discard features 6.4 Dependence on target model complexity that are not needed to solve the pretext task during train- ing. These models, therefore, mismatch with ill-posed tar- In Fig. 4 and Table 1 we observe a OFM spike early in get tasks. For example, rotation prediction discards features training for more complex target models. This spike occurs corresponding to the hue while it learns much about the probably because nonlinear target models make better orientation of the object. sense of specific random features at pretext task initial- ization, in contrast to the linear target model. Besides early 6.7 Applying our metrics to ResNet models spikes, mismatches tend to decrease when we add com- plexity to the target model. A model with increased non- In Fig. 6, we apply our metrics to ResNet models for linearity has more freedom to disentangle representations, several pretext and target tasks. For contrastive learning we Fig. 5 (Bottom) Pretext losses of our model trained for color restoration (CCAE), prediction rotations (RCAE), and contrastive learning (SCLCAE). (Top) Best target losses of linear models trained for the different prediction tasks of 3dshapes 123 Neural Computing and Applications Table 2 cSM3, MOFM and MM3 on the 3dshapes dataset CAE DCAE CCAE RCAE SCLCAE cSM3 MOFM cSM3 MOFM cSM3 MOFM cSM3 MOFM MM3 cSM3 MOFM MM3 Floor_hue 0.01 0.95 0.00 1.28 0.02 0.00 56.68 1 44.67 28.18 268.27 48.38 Wall_hue 0.02 32.03 0.00 24.43 0.10 0.00 25.17 1 7.80 0.29 0.46  76.40 Object_hue 0.38 22.71 0.43 24.55 1.55 0.63 59.65 1 40.1 2.87 8.69 73.17 Scale 0.41 0.00 0.27 0.00 0.10 0.00 2.60 0.13 31.78 2.43 0.00 44.80 Shape 0.07 0.00 0.08 0.00 0.03 0.00 0.20 0.06  2.48 1.67 2.16 67.54 Orientation 0.00 0.00 0.00 0.00 0.23 0.00 0.48 0.00 22.26 2.50 6.68 9.11 Average 0.15 9.28 0.13 8.21 0.34 0.11 24.13 1 24.02 6.32 47.71  53.23 The smallest mismatches for each setup are printed in bold SM3 and MM3 are measured on the target task accuracy. Values are obtained by 5-fold cross-validation. We show the stability of each measurement in Tables 8, 9 and 10 of Appendix B Fig. 6 OFM for different pretext tasks trained with a ResNet18 model as a backbone. The mismatches are shown for the entire training process observe a small OFM for Cifar10 and Cifar100, which 7 Future work occurs late in training after pretext model convergence. However, when we use contrastive learning as pretext task In future work, our metrics can be used to create, tune and for fine-grained tumor detection on the PCam dataset, we evaluate (self-supervised) representation learning methods observe a mismatch before pretext model convergence. For for different target tasks and datasets. These metrics make the well-known rotation prediction pretext task, we it possible to quantify the extent to which a pretext task oberved a high mismatch on Cifar10 classification early in matches a target task, and to determine whether the pretext pretext training. In Table 3, we show the corresponding task learns the right kind of representation throughout the mismatches measured until pretext model convergence. entire training process. This enables a comparison of methods on benchmarks across different pretext tasks and 123 Neural Computing and Applications Table 3 Mismatches of ResNets with convergence criterium draft and Jordi Gonzàlez for his much-appreciated feedback on this work. ResNet Funding Open Access Funding provided by Universitat Autonoma de ACC cSM3 MOFM MM3 Barcelona. RCAE (Cifar10) þ1:80 þ1:74 þ4:42 þ0:75 54:642:01 3:983:60 4:873:11 31:820:68 Data availability All datasets are available in the public domain. þ0:44 þ0:44 þ1:00 SCLCAE (PCam) 96:250:23 0:370:37 0:860:60 53:26þ0:52 0:38 Code availability Our implementation is available at https://github. ACC stands for the best accuracy on the target task of all target com/BonifazStuhr/OFM. models trained on the pretext model. cSM3 corresponds to the accuracy we would lose when we naively train the target model after pretext model convergence. Values are obtained by 5-fold cross- Declarations validation. The mismatches are measured with a convergence criterium Conflicts of interest Bonifaz Stuhr and Jürgen Brauer are members of the University of Applied Sciences Kempten. Bonifaz Stuhr is a member of the Autonomous University of Barcelona. models. The dependencies of the objective function mis- Open Access This article is licensed under a Creative Commons match on different parts of the self-supervised setup (e.g., Attribution 4.0 International License, which permits use, sharing, representation size) can be explored by future work in more adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the detail, to further evaluate our findings and to create pretext source, provide a link to the Creative Commons licence, and indicate tasks and model architectures that are robust against mis- if changes were made. The images or other third party material in this matches. Our metrics are defined for setups where the article are included in the article’s Creative Commons licence, unless target models are trained on pretext model representations indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended in general. Therefore, they can also be applied to other use is not permitted by statutory regulation or exceeds the permitted representation learning areas such as supervised, semi-su- use, you will need to obtain permission directly from the copyright pervised, few-shot, or biological plausible representation holder. To view a copy of this licence, visit http://creativecommons. learning. org/licenses/by/4.0/. References 8 Conclusion 1. Ackley DH, Hinton GE, Sejnowski TJ (1985) A learning algo- In this work, we have used the linear evaluation protocol as rithm for boltzmann machines. Cogn Sci a basis to define and discuss metrics to measure the metrics 2. Arora S, Khandeparkar H, Khodak M, Plevrakis O, Saunshi N (2019) A theoretical analysis of contrastive unsupervised repre- mismatch and the objective function mismatch. With soft sentation learning. CoRR and hard versions of our metrics, we collected evidence of 3. Asano YM, Rupprecht C, Vedaldi A (2020) A critical analysis of how these mismatches relate to the pretext model’s repre- self-supervision, or what we can learn from a single image. In: sentation size, target model complexity, pretext and target International conference on learning representations 4. Bachman P, Hjelm RD, Buchwalter W (2019) Learning repre- augmentations as well as pretext, and target task types. sentations by maximizing mutual information across views. In: Furthermore, we observe that the epoch of the target task Advances in neural information processing systems peak performance varies strongly for different datasets and 5. Burgess C, Kim H (2018) 3d shapes dataset. https://github.com/ pretext tasks. This highlights the importance of the protocol deepmind/3dshapes-dataset/ 6. Cao X, Chen BC, Lim SN (2019) Unsupervised deep metric and shows that comparing approaches after a fixed number learning via auxiliary rotation loss. CoRR of epochs does not yield the entire picture of their capa- 7. Caron M, Bojanowski P, Joulin A, Douze M (2018) Deep clus- bility. Our protocols make it possible to define benchmarks tering for unsupervised learning of visual features. In: Proceed- across different target tasks, where the goal is not to mis- ings of the European Conference on Computer Vision (ECCV) 8. Charte D, Charte F, del Jesus MJ, Herrera F (2020) An analysis match with the target metrics while achieving the best on the use of autoencoders for representation learning: funda- possible performance. mentals, learning task case studies, explainability and challenges. Neurocomputing Supplementary Information The online version contains 9. Chen T, Kornblith S, Norouzi M, Hinton GE (2020) A simple supplementary material available at https://doi.org/10.1007/s00521- framework for contrastive learning of visual representations. In: 022-07031-9. International conference on machine learning 10. Chen T, Zhai X, Ritter M, Lucic M, Houlsby N (2019) Self- Acknowledgments Really kindly, we want to thank our colleagues supervised gans via auxiliary rotation loss. In: Proceedings of the from the University of Applied Sciences Kempten and the Autono- IEEE conference on computer vision and pattern recognition mous University of Barcelona for the helpful discussions about this 11. Chen X, Fan H, Girshick RB, He K (2020) Improved baselines topic. Especially we want to thank Markus Klenk for proofreading the with momentum contrastive learning. CoRR 123 Neural Computing and Applications 12. Doersch C, Gupta A, Efros AA (2015) Unsupervised visual 33. Krizhevsky A (2009) Learning multiple layers of features from representation learning by context prediction. In: Proceedings of tiny images. Tech Rep, University of Toronto the IEEE international conference on computer vision 34. Larsson G, Maire M, Shakhnarovich G (2016) Learning repre- 13. Doersch C, Zisserman A (2017) Multi-task self-supervised visual sentations for automatic colorization. In: European conference on learning. In: Proceedings of the IEEE international conference on computer vision computer vision 35. Larsson G, Maire M, Shakhnarovich G (2017) Colorization as a 14. Donahue J, Simonyan K (2019) Large scale adversarial repre- proxy task for visual understanding. In: Proceedings of the IEEE sentation learning. In: Advances in neural information processing conference on computer vision and pattern recognition systems 36. Li J, Zhou P, Xiong C, Socher R, Hoi SCH (2020) Prototypical 15. Fernando B, Bilen H, Gavves E, Gould S (2017) Self-supervised contrastive learning of unsupervised representations. CoRR video representation learning with odd-one-out networks. In: 37. Lin F, Xu H, Li H, Xiong H, Qi GJ (2019) Aetv2: autoencoding Proceedings of the IEEE conference on computer vision and transformations for self-supervised representation learning by pattern recognition minimizing geodesic distances in lie groups. CoRR 16. Gidaris S, Singh P, Komodakis N (2018) Unsupervised repre- 38. Linsker R (1988) Self-organization in a perceptual network. sentation learning by predicting image rotations. CoRR Computer 17. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, 39. Locatello F, Bauer S, Lucic M, Gelly S, Schölkopf B, Bachem O Ozair S, Courville A, Bengio Y (2014) Generative adversarial (2018) Challenging common assumptions in the unsupervised nets. In: Advances in neural information processing systems learning of disentangled representations. CoRR 18. Goyal P, Mahajan D, Gupta A, Misra I (2019) Scaling and 40. Lorena AC, Garcia LP, Lehmann J, Souto MC, Ho TK (2019) benchmarking self-supervised visual representation learning. In: How complex is your classification problem? a survey on mea- Proceedings of the IEEE international conference on computer suring classification complexity. ACM Computing Surveys vision (CSUR) 19. Hadsell R, Chopra S, LeCun Y (2006) Dimensionality reduction 41. Metz L, Maheswaranathan N, Cheung B, Sohl-Dickstein J (2018) by learning an invariant mapping. In: 2006 IEEE computer Learning unsupervised learning rules. CoRR society conference on computer vision and pattern recognition 42. Newell A, Deng J (2020) How useful is self-supervised pre- (CVPR’06) training for visual tasks? In: Proceedings of the IEEE/CVF 20. Haque KN, Rana RK, Schuller B (2020) Guided generative Conference on Computer Vision and Pattern Recognition adversarial neural network for representation learning and high 43. Noroozi M, Favaro P (2016) Unsupervised learning of visual fidelity audio generation using fewer labelled audio data. CoRR representations by solving jigsaw puzzles. In: European Confer- 21. He K, Fan H, Wu Y, Xie S, Girshick R (2020) Momentum ence on Computer Vision. Springer contrast for unsupervised visual representation learning. In: 44. Oliver A, Odena A, Raffel CA, Cubuk ED, Goodfellow I (2018) Proceedings of the IEEE/CVF conference on computer vision and Realistic evaluation of deep semi-supervised learning algorithms. pattern recognition In: Advances in neural information processing systems 22. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for 45. Palacio-Niño J, Berzal F (2019) Evaluation metrics for unsu- image recognition. In: Proceedings of the IEEE conference on pervised learning algorithms. CoRR computer wision and pattern recognition 46. Patacchiola M, Storkey AJ (2020) Self-supervised relational 23. Hendrycks D, Mazeika M, Kadavath S, Song D (2019) Using reasoning for representation learning. CoRR self-supervised learning can improve model robustness and 47. Pathak D, Krahenbuhl P, Donahue J, Darrell T, Efros AA (2016) uncertainty. In: Advances in neural information processing Context encoders: Feature learning by inpainting. In: Proceedings systems of the IEEE conference on computer vision and pattern 24. Hjelm RD, Fedorov A, Lavoie-Marchildon S, Grewal K, recognition Trischler A, Bengio Y (2019) Learning deep representations by 48. Patrick M, Asano YM, Kuznetsova P, Fong R, Henriques JF, mutual information estimation and maximization. CoRR Zweig G, Vedaldi A (2020) Multi-modal self-supervision from 25. Hochreiter S, Younger AS, Conwell PR (2001) Learning to learn generalized data transformations. CoRR using gradient descent. In: International conference on artificial 49. Qi GJ, Zhang L, Chen CW, Tian Q (2019) Avt: Unsupervised neural networks learning of transformation equivariant representations by 26. Jing L, Tian Y (2020) Self-supervised visual feature learning with autoencoding variational transformations. In: Proceedings of the deep neural networks: a survey. IEEE Trans Pattern Anal Mach IEEE international conference on computer vision Intel. https://doi.org/10.1109/TPAMI.2020.2992393 50. Radford A, Metz L, Chintala S (2015) Unsupervised represen- 27. Khodadadeh S, Boloni L, Shah M (2019) Unsupervised meta- tation learning with deep convolutional generative adversarial learning for few-shot image classification. In: Advances in neural networks. CoRR information processing systems 51. Rumelhart DE, McClelland JL (1987) Learning internal repre- 28. Kim D, Cho D, Kweon IS (2019) Self-supervised video repre- sentations by error propagation. American Association for the sentation learning with space-time cubic puzzles. In: Proceedings Advancement of Science of the AAAI conference on artificial intelligence 52. Schmidhuber J (1987) Evolutionary principles in self-referential 29. Kim D, Cho D, Yoo D, Kweon IS (2018) Learning image rep- learning on learning how to learn: The meta-meta-meta...-hook. resentations by completing damaged jigsaw puzzles. In: 2018 Diploma thesi, Technische Universitat Munchen IEEE Winter Conference on Applications of Computer Vision 53. Schmidhuber J (1990) Making the world differentiable: On using (WACV) self-supervised fully recurrent neural networks for dynamic 30. Kingma DP, Ba J (2014) Adam: a method for stochastic opti- reinforcement learning and planning in non-stationary environ- mization. CoRR ments. Tech rep, Technische Universitat Munchen 31. Kingma DP, Welling M (2014) Auto-encoding variational bayes. 54. Schmidhuber J (1995) On learning how to learn learning strate- CoRR gies. Tech rep, Technische Universitat Munchen 32. Kolesnikov A, Zhai X, Beyer L (2019) Revisiting self-supervised 55. Shukla A, Petridis S, Pantic M (2020) Does visual self-supervi- visual representation learning. In: Proceedings of the IEEE con- sion improve learning of speech representations? CoRR ference on computer vision and pattern recognition 123 Neural Computing and Applications 56. Srinivas A, Laskin M, Abbeel P (2020) Curl: contrastive unsu- 66. Wen Z (2020) Convergence of end-to-end training in deep pervised representations for reinforcement learning. CoRR unsupervised contrasitive learning. CoRR 57. Srivastava N, Mansimov E, Salakhudinov R (2015) Unsupervised 67. Wolf S, Hamprecht FA, Funke J (2020) Instance separation learning of video representations using lstms. In: International emerges from inpainting. CoRR conference on machine learning 68. Wu M, Zhuang C, Mosse M, Yamins DLK, Goodman ND (2020) 58. Stuhr B, Brauer J (2019) Csnns: unsupervised, backpropagation- On mutual information in contrastive learning for visual repre- free convolutional neural networks for representation learning. sentations. CoRR In: 2019 18th IEEE International Conference On Machine 69. Xie S, Gu J, Guo D, Qi CR, Guibas L, Litany O (2020) Point- Learning And Applications (ICMLA) contrast: unsupervised pre-training for 3d point cloud under- 59. Su JC, Maji S, Hariharan B (2019) When does self-supervision standing. In: European conference on computer vision improve few-shot learning? CoRR 70. Zhai X, Puigcerver J, Kolesnikov A, Ruyssen P, Riquelme C, 60. Veeling BS, Linmans J, Winkens J, Cohen T, Welling M (2018) Lucic M, Djolonga J, Pinto AS, Neumann M, Dosovitskiy A et al Rotation equivariant cnns for digital pathology. In: International (2019) A large-scale study of representation learning with the conference on medical image computing and computer-assisted visual task adaptation benchmark. CoRR intervention 71. Zhan X, Xie J, Liu Z, Ong YS, Loy CC (2020) Online deep 61. Vincent P, Larochelle H, Bengio Y, Manzagol PA (2008) clustering for unsupervised representation learning. In: Proceed- Extracting and composing robust features with denoising ings of the IEEE/CVF conference on computer vision and pattern autoencoders. In: Proceedings of the 25th international confer- recognition ence on machine learning 72. Zhang L, Qi GJ, Wang L, Luo J (2019) Aet vs. aed: Unsupervised 62. Vondrick C, Shrivastava A, Fathi A, Guadarrama S, Murphy K representation learning by auto-encoding transformations rather (2018) Tracking emerges by colorizing videos. In: Proceedings of than data. In: Proceedings of the IEEE conference on computer the European Conference on Computer Vision (ECCV) vision and pattern recognition 63. Voynov A, Morozov S, Babenko A (2020) Big gans are watching 73. Zhang R, Isola P, Efros AA (2016) Colorful image colorization. you: Towards unsupervised object segmentation with off-the- In: European conference on computer vision shelf generative models. CoRR 64. Wallace B, Hariharan B (2020) Extending and analyzing self- Publisher’s Note Springer Nature remains neutral with regard to supervised learning across domains. CoRR jurisdictional claims in published maps and institutional affiliations. 65. Weinberger KQ, Blitzer J, Saul LK (2006) Distance metric learning for large margin nearest neighbor classification. In: Advances in neural information processing systems 123