This is the accepted version of the journal article: Katakol, Sudeep; Elbarashy, Basem; Herranz, Luis; [et al.]. «Distributed learning and inference with compressed images». IEEE transactions on image processing, Vol. 30 (Dec. 2021), p. 3069-3083. 15 pàg. DOI 10.1109/TIP.2021.3058545 This version is available at https://ddd.uab.cat/record/274821 under the terms of the license 1 Distributed Learning and Inference with Compressed Images Sudeep Katakol, Basem Elbarashy, Luis Herranz, Joost van de Weijer, and Antonio M. López, Member, IEEE Abstract—Modern computer vision requires processing large possible in resource-limited devices. Thus, training is often amounts of data, both while training the model and/or during performed in a centralized server, which also allows using inference, once the model is deployed. Scenarios where images data captured by multiple devices to train better models (e.g. are captured and processed in physically separated locations are increasingly common (e.g. autonomous vehicles, cloud computing, a fleet of autonomous cars). In this case, training and testing smartphones). In addition, many devices suffer from limited take place in two physically separated locations, i.e. server resources to store or transmit data (e.g. storage space, channel and device, respectively. In other cases, such as in mobile capacity). In these scenarios, lossy image compression plays cloud computing, the data is captured by the device, while the a crucial role to effectively increase the number of images inference takes place in a server. collected under such constraints. However, lossy compression entails some undesired degradation of the data that may harm One important requirement in these scenarios is that, at the performance of the downstream analysis task at hand, since some point, the visual data needs to be transmitted from the important semantic information may be lost in the process. device to the server. Fig. 1a shows an archetypal scenario of Moreover, we may only have compressed images at training autonomous driving, where each vehicle of the fleet captures time but are able to use original images at inference time and encodes data and transmits it to the server. The server (i.e. test), or vice versa, and in such a case, the downstream model suffers from covariate shift. In this paper, we analyze this decodes the data and uses it for training the analysis models. phenomenon, with a special focus on vision-based perception The trained models are then deployed to autonomous vehicles, for autonomous driving as a paradigmatic scenario. We see where they perform inference. Captured data often requires to that loss of semantic information and covariate shift do indeed be annotated by humans in order to train supervised models, exist, resulting in a drop in performance that depends on the which adds to the reasons to process the data in a server. compression rate. In order to address the problem, we propose dataset restoration, based on image restoration with generative The captured data can be stored on-board in a storage adversarial networks (GANs). Our method is agnostic to both device and physically delivered to the server, or directly the particular image compression method and the downstream transmitted through a communication channel. In either case, task; and has the advantage of not adding additional cost to the storage space or channel capacity are constraints that condition deployed models, which is particularly important in resource- limited devices. The presented experiments focus on semantic the amount of collected samples in practice, and effective segmentation as a challenging use case, cover a broad range collection requires data compression to exploit the limited of compression rates and diverse datasets, and show how our storage and communication resources efficiently. method is able to significantly alleviate the negative effects of The amount of data captured (possibly from multiple cam- compression on the downstream visual task. eras) can be enormous, requiring high compression rates with Index Terms—Image compression, image restoration, gener- lossy compression. However, this entails a certain degradation ative adversarial networks, deep learning, autonomous driving. in the images, which depends on the bitstream rate (the lower the rate, the higher the degradation). In this paper, we study the impact of such degradation on the performance of the I. I NTRODUCTION downstream analysis task. At times, the degradation affects only one of the training and test data. For instance, in Fig 1a, M ODERN intelligent devices such as smartphones, au- tonomous vehicles and robots are equipped with high- quality cameras and powerful deep neural networks that enable training data is degraded while test data (on-board) can be accessed without degradation. advanced on-board visual analysis and understanding. These When training data is compressed and test is not (or vice large models are trained with a large amount of data and versa), a first effect we observe is covariate shift (i.e. the require powerful hardware resources (e.g. GPUs). These mod- training and test data are sampled from different distributions). els also require days or even weeks to train, which is not For instance, the first column of Fig. 1b represents the original captured images, while the second represents the compressed S. Katakol is with the Department of Computer Science & Information images (i.e. reconstructed1 ). A clear difference in terms of Systems and the Department of Mathematics, BITS Pilani, KK Birla Goa lack of details and blurred textures is observed, which causes Campus, India, 403726. This work was done during an internship at Computer Vision Center, Barcelona. covariate shift (e.g. original for test, compressed for training E-mail: skatakol@cvc.uab.es in the example of Fig 1a). A possible solution to this problem is compressing both training and test data at the same rate. For B. Elbarashy, L. Herranz, J. van de Weijer and A. M. López are with the Computer Vision Center, Universitat Autònoma de Barcelona, 08193 Bellaterra, Spain. J. van de Weijer and A. M. López are also associated with 1 When referring to data used in the downstream tasks, compressed images the Computer Science Dept. at Universitat Autònoma de Barcelona. will implicitly refer to the reconstructed images after the compression decoder. 2 Encoder Training Analysis Encoder Channel Decoder Data module 80 Encoder Segmentation mIoU (%) Compression Capture 75 Restoration Annotation Deployment Test 70 Analysis module Analysis module 65 Analysis module Captured Compressed Restored OO CO CC RO Train/test configuration (a) (b) (c) Fig. 1. Problem statement and proposed approach: (a) data collection using lossy compression makes training (top) and test data (bottom) different, (b) differences between test and training data are alleviated using adversarial restoration, and (c) drop in segmentation performance due to lossy compressed training data (CO/CC) and benefit from the proposed restoration method (RO). the autonomous driving scenario of Fig. 1a, this would mean CC configurations. As a result, we propose dataset restoration, deploying an image compressor in the car (including encoder an effective approach based on image restoration using gener- and decoder) and performing inference on the reconstructed ative adversarial networks (GANs) [1]. Dataset restoration is images. While this approach alleviates the covariate shift, it is applied to the images in the training set without modifying the not always effective and also increases the computational cost test images, effectively alleviating the covariate shift, while in the on-board system. keeping the test data semantically informative. In this case, The degradation caused by lossy compression not only in- we show that the configuration restored-original (RO) does duces covariate shift, but can also harm the performance of the improve performance over the baselines (see Fig. 1c). An downstream task through the means of semantic information additional advantage is that there is no computational cost loss. Here, semantic information refers to the information that penalty nor additional hardware or software requirements in is relevant to solve a particular downstream task and it can be the deployed on-board system (in contrast to compressing the lost during the process of compression. Semantic information test data). Note also that our approach is generic and inde- is task-dependent and its loss is typically irreversible. For pendent of the particular compression (deep or conventional) example, the actual plate number WAF BA 747 in the second used to compress the images. column of Fig. 1b is lost in the process of compression, and Adversarial restoration decreases the covariate shift by cannot be recovered. However, if the task is car detection, hallucinating texture patterns that resemble those lost during the actual plate number is not necessarily relevant semantic compression while removing compression artifacts, both of information. which contribute to the covariate shift. The distribution of In this paper, we study the effect of compression on down- restored images is closer to the distribution of original images stream analysis tasks (focusing on semantic segmentation) and thus the covariate shift is lower. Fig. 1b shows an example under different configurations, which in turn can be related where the trees have lost their texture and appear essentially as to real scenarios. We observe that both covariate shift and blurred green areas. A segmentator trained with these images semantic information loss indeed result in a performance drop will expect trees to have this appearance, but during test (see Fig. 1c2 ) compared to training and test with original they appear with the original texture and details of leaves images (configuration OO). The performance depends on the and branches, which leads to poor performance. The restored compression rate and the particular training/test configuration. image has textures that resemble real trees and contains less For instance, in the configuration of the autonomous driving compression artifacts, which makes its distribution closer to scenario of Fig. 1a, compressing the test data prior to inference that of the actual test images, contributing to a significant (we refer to this approach as compression before inference, improvement in downstream performance (see Fig 1c). Note and corresponds to the training/test configuration compressed- that adversarial restoration cannot recover certain semantic in- compressed, or CC for short) degrades the performance more formation. This example also illustrates the effect on semantic than using the original data (configuration CO), showing information. The license plate appears completely blurred due that it is preferable to keep the test data more semantically to compression. Note that adversarial restoration can recover informative than correcting the covariate shift. the texture of digits (or even hallucinate random digits), which The previous result also motivates us to explore whether can be useful to improve car segmentation, but the original there exists a solution that improves over the baseline CO and plate number is lost (i.e. semantic information), which makes it impossible to perform license plate recognition at that 2 Segmentation performance is measured as the mean Intersection over Union (mIoU), which is the ratio between the correctly predicted pixels and compression rate. union of predicted and ground truth pixels, averaged across every category. In summary, our contributions are as follows: 3 • Systematic analysis of training/test configurations with C. Image restoration compression and relation of downstream performance Image restoration involves the process of improving the with rate, semantic information loss and covariate shift. quality of degraded images. Restoration methods can be • Dataset restoration, a principled method based on our the- grouped into denoising [24], deblurring [25], [26], super- oretical analysis, to improve downstream performance in resolution [27], compression artifact removal [28], etc. de- on-board analysis scenarios. This method is task-agnostic pending on the kind of degradation, although they share and can be used alongside multiple image compression many similarities. Lately, deep learning methods have been methods. It also does not increase the inference time and successful for image restoration tasks. Some of these methods memory requirements of the downstream model. can be applied to any degradation [29], [30] while others II. R ELATED W ORK are specific to the degradation (deblurring [31], [32], super- resolution [33], [34], denoising [35], [36] and compression A. Lossy compression artifact removal [37]). More recently, image restoration algo- A fundamental problem in digital communication is the rithms based on generative adversarial networks (GANs) have transmission of data as binary streams (bitstreams) under become popular owing to their improved performance (super- limited capacity channels [2], [3], a problem addressed by resolution [38], [39], compression artifact removal [40], [41] data compression. Often, practical compression ratios are and deblurring [42]). achievable only with lossy compression, i.e. a certain loss A compressed image can be processed using a restoration with respect to the original data is tolerated. Traditional lossy method before using it for inference to improve its perfor- compression algorithms for images typically use a DCT or a mance; although our analysis reveals that this is a sub-optimal wavelet transform to transform the image into a compact rep- approach. Galteri et al. [40] propose a GAN-based restoration resentation, which is simplified further to achieve the desired network to correct JPEG compression artifacts. They also bitrate. Examples of lossy image compression algorithms are evaluate different restoration algorithms on the basis of the JPEG [4], JPEG 2000 [5], [6], and BPG [7]. BPG is the current performance of restored images on a trained object detection state-of-the-art and is based on tools from the HEVC video network. They show that their GAN-based algorithm performs coding standard [8]. better than other methods compared in the paper. Our analysis Recently, deep image compression [9]–[14] has emerged provides an explanation for this observation. as a powerful alternative to the traditional algorithms. These methods also use a transformation based approach like the tra- ditional methods, but use deep neural networks to parameterize D. Domain adaptation. the transformation [9]. The parameters of the networks are Domain adaptation [43] is a problem motivated by the learned by optimizing for a particular rate-distortion tradeoff lack of sufficient annotations. Typically, domain adaptation on a chosen dataset. Mean Scale Hyperprior (MSH) [13], a methods leverage the abundant annotated data available from deep image compression method based on variational autoen- a different yet related domain (called as the source domain) coders and BPG are used as representative methods of deep to improve performance on the domain of interest (target learning based and traditional image compression respectively. domain), where there is a lack of annotated data. Examples of source and target domains include synthetic images vs real im- B. Visual degradation and deep learning ages, images in the wild vs images on a webpage, etc. Domain A loss in the quality of images can occur through many adaptation methods can be divided into unsupervised [44], factors including blur, noise, downsampling and compres- [45], semi-supervised [46] and supervised [47]–[51] categories sion. Researchers have reported a drop in task performance depending on the quantity of available data (and annotations) of convolutional neural networks (CNN) models when such in the target domain. Approaches for domain adaptation can degradations are present in the test images [15]–[17]. Fur- be categorized into latent feature alignment using autoen- ther, numerous methods have been proposed to make these coders [47], [48], adversarial latent feature alignment [49]– CNN models robust to degradations [16], [18], [19]. These [52] and pixel-level adversarial alignment [45], [51]. approaches include forcing adversarial robustness during train- The scenario when only compressed images are available ing [16], modifying and retraining the network [18], and using at training time, with original images available at test time is an additional network altogether [19]. related to domain adaptation. Dataset restoration, our proposed While the aforementioned works target robustness across method for this scenario, corrects the covariate shift and degradations, there have been studies focusing exclusively on domain adaptation algorithms account for domain shift in compression as well. These include [20] (on the deep com- some form. Probably, the closest domain adaptation method to pression method [12]), [21] (JPEG) and [22] (both deep [23] dataset restoration is an unsupervised method that addresses and JPEG). Unlike the previous methods, these works use alignment only at pixel-level [45]. However, an important the compressed images (in some form) for training the deep distinction is that domain adaptation tackles the problems models and thus obtain a better performance on compressed arising due to lack of annotations for the images in the target images. Moreover, [20] and [21] encode the images using the domain, while for us the concern lies in the non-availability of compressors and the deep networks are trained to predict the images themselves. Thus, we study the effectiveness of dataset task output using the encoded representation directly, resulting restoration using an external dataset for training and also by in faster inference. varying the number of original training images. 4 III. L EARNING AND INFERENCE WITH COMPRESSED However, covariate shift exists in the other two configurations, IMAGES namely CO and OC (see Table I). A. Problem definition The degradation due to lossy compression can be observed clearly in Fig. 1b, when comparing the original captured image We are concerned with downstream understanding tasks and the image after compression. This also gives an idea of where we want to infer from an input image x ∼ pX (x), the difference between the original domain and the domain the corresponding semantic information y ∼ pY |X (y). In induced by compression. It has images with lesser details the rest of the paper, we will assume that y is a semantic which also suffer from blurring and coding artifacts. More segmentation map, but our approach can also be applied to examples are shown in Fig. 2 for the two compression methods other semantic inference tasks, such as image classification or (MSH and BPG), with the images compressed at a similar object detection. The objective is to find a parametric mapping rate. It can be seen that degradations are consistent yet with φ : x n7→ y by supervised  learning from o a training dataset (1) (1)  (N ) (N ) some differences (e.g. blocky artifacts for BPG, more blurred Xtr = xtr , ytr , . . . , xtr , ytr , where each image in MSH). x(i) has a corresponding ground truth annotation y(i) . The mapping is typically implemented as a deep neural network. The performance n of the resulting  model  is evaluated o on a test set Xts = (1) (1) (M ) (M ) xts , yts , . . . , xts , yts . Under C. Semantic information loss conventional machine learning assumptions, Xtr ∼ pX (x) Covariate shift explains how compression impacts the down- and Xts ∼ pX (x), i.e. both training and test sets are sampled stream task when data at training and test time are compressed from the same underlying distribution pX . unequally. Another factor that impacts task performance arises In our setting, we consider that Xtr and/or Xts undergo a from compression, resulting in semantic information loss. By certain degradation ψ : x 7→ x̂. In our case, the degradation the semantic information present in an image we refer only to is related with the lossy compression process necessary to what is relevant to the downstream task. Thus, by definition, transmit the image to the remote location where the actual semantic information loss is task dependent. Continuing with training or inference takes place; and so we have x̂ = ψ (x) = our example from the introduction (Fig. 1b), the letters in the g (f (x)), where f (x) is the image encoder, g (z) is the image license plate of the car plays little to no role in establishing decoder3 and z is the compressed bitstream. The result x̂ is the presence of a car in the image. Thus, the exact letters are the reconstructed image, which follows a new distribution pX̂ not relevant semantic information for the task of car detection. of degraded images, i.e. x̂ ∼ pX̂ (x). Note that parallels can However, if the task is license plate recognition, the letters are be drawn from the arguments in this section for other image an integral part of semantic information. Compression causes degradations such as blur, downsampling, noise, color and semantic information loss as it makes the compressed image illumination changes, etc. devoid of some semantic attributes present in the original Lossy compression is characterized by the distortion image. The loss of letters on the plate in the compressed image D (x, x̂) of the reconstructed image and the rate R (z) of the (Fig. 1b) is evidence of semantic information loss (when the compressed bitstream. The encoder and decoder are designed task is license plate detection). to operate around a particular rate-distortion (R-D) tradeoff λ, Further evidence of semantic information loss can be found either by expert crafting in conventional image compression, in Fig. 2, since the degradation often removes details and or by directly optimizing parameters of a deep neural network. textures, blends small objects together via blur and lack of contrast, and introduces confusing artifacts, preventing us from B. Covariate shift recognizing small objects at all (e.g. individual pedestrians), The covariate shift problem precisely occurs when the and making larger objects more difficult to recognize due to underlying distributions of training and test data differ, i.e. the loss of discriminative details and textures (e.g. tree leaves). Xtr ∼ pXtr and Xts ∼ pXts with pXtr 6= pXts . This leads to Only in retrospective, after observing the original undistorted sub-optimal performance because the model is evaluated on a crop, we can infer the small objects in the distorted image. data distribution different from the one it was optimized for. Similarly, a semantic segmentation model will struggle to While covariate shift is often found in machine learning (e.g. recognize them, or directly fail when the semantic information training with synthetic data and evaluating on real), in our case, has disappeared completely (e.g. license plate number). this problem is a consequence of lossy compression and it in- Let Y be a random variable that represents the semantic creases severely as the rate decreases. The drop in performance information in the original image, X. For instance, if the task is related to the degree of covariate shift, which could be seen is semantic segmentation, Y would take values from the set as the divergence between distributions d (pXtr , pXts ). In the of semantic maps of images. Mathematically, we formulate conventional machine learning setting without compression, semantic information loss, S, in the compressed images, there is no covariate shift, since Xtr ∼ pX and Xts ∼ pX , nor X̂, using mutual information, I, as follows: SY (X, X̂) = when both training and test set are compressed with the same I(X, Y ) − I(X̂, Y ). Predictably, SY (X, X̂) is non-negative, method and at the same rate, since Xtr ∼ pX̂ and Xts ∼ pX̂ . since X̂ is produced from X via the map ψ and thus, we have I(X, Y ) ≥ I(X̂, Y ) as a consequence of the data processing 3 We only consider lossy compression, since in lossless compression x̂ = x. inequality. 5 Fig. 2. Effects of compression and restoration (from left to right): captured image, compressed (BPG), restored (BPG), compressed (MSH), restored (MSH). The brightness of the image crops have been slightly enhanced to improve visibility. TABLE I Training T RAINING / TEST CONFIGURATIONS Analysis Capture Data module (HQ) Distribution Inf. loss Cov. Examples Annotation Config. Train Test Train Test shift Encoder Deployment OO pX pX No No No Most machine learning On-board analysis Encoder Channel Decoder Analysis CO pX̂ pX Yes No Large module (autonomous cars, drones) Cloud computing, Encoder OC pX pX̂ No Yes Large distributed automotive perception Test Compression before CC pX̂ pX̂ Yes Yes No training/inference Fig. 3. Example of OC configuration: mobile cloud computing with inference OR pX pX̄ Yes No Medium Image restoration on compressed test images, and high quality training images. RO pX̄ pX No Yes Medium Dataset restoration of analysis (e.g. predicted class, bounding box, segmentation D. Training/test configurations, application scenarios and re- map). Fig. 3 illustrates the paradigmatic scenario of (mobile) lated work cloud computing [53]–[55]. Another example of OC config- uration is distributed automotive perception [22], where the Now, we focus on several training/test configurations and sensor module compresses the captured image and transmits provide examples of real world scenarios (summarized in it through the automotive bus system to the perception module Table I). A configuration is defined by the pair (Xtr , Xts ), where the downstream tasks are performed. with Xi ∼ pX represented as O and Xi ∼ pX̂ represented as C. Thus, the conventional machine learning setting (i.e. The configuration CC appears in the previous scenario when without compression) corresponds to OO, and the configu- training images are also compressed, and at the same rate as ration of Fig. 1a is CO, since Xtr ∼ pX̂ and Xts ∼ pX . test images. In this case, both training and test images suffer The former does not suffer from semantic information loss from semantic information loss, but there is no covariate shift nor covariate shift, while the latter does suffer from both. The since both are sampled from the same pX̂ . CO configuration can also be generalized to other scenarios Compression before training. We can remove the covariate involving on-board analysis4 where data capture and inference shift from the configuration OC by transforming it into CC. takes place in the device and the training in a server (e.g. This can be achieved by compressing the training data at autonomous cars, unmanned aerial vehicles and other robotic the same rate and we refer to this adaptation approach as devices). compression before training (see Fig. 4a). Naturally, since CC The configuration OC involves training in the server with the is unaffected by covariate shift unlike OC, we expect the model original images, while inference is performed with compressed with configuration CC to outperform configuration OC. images, leading to semantic information loss and covariate As a downside of the process of compression before train- shift. This is often the case when the capturing device has ing, semantic information loss is additionally introduced into limited resources to perform complex analysis (e.g. smart- the training set. However, the presence or absence of semantic phone), but can compress and send the content through a information loss in the test images is a major factor, while it communication channel, and then receive back the results is not the case with the training images. The segmentation network is trained using the entire set and if some class 4 Often called on-board perception, but we prefer on-board analysis to avoid information is lost in a particular training image, its presence confusion later. in other training images can compensate for it. Thus, we can 6 usually get away with introducing some semantic information Server loss in the training set. In contrast, as the segmentation network is evaluated on individual images in the test set, Encoder Decoder Adapted Dataset dataset the performance suffers critically by the presence of semantic information loss. Compression before inference. Similarly, we can also Adapted Analysis module transform configuration CO into CC by compressing the dataset test images. We refer to this process as compression before (a) inference (see Fig. 4b). While this process allows us to correct Car the covariate shift due to compression, it also introduces Analysis Encoder Decoder module semantic information loss at test time. The introduction of semantic information loss in the test is critical and can cause the performance of configuration CC to be even worse (b) Server than the configuration CO at times (as shown in Fig. 1c). Moreover, compression before inference requires installing a Dataset Restoration (restoration) model full compression encoder and decoder module on-board prior to the downstream task, resulting in a significant computational penalty in the deployed system. Restoraton Analysis Dataset Restored module model (analysis) dataset IV. DATASET RESTORATION A. Proposed approach (c) Motivated by the two limitations mentioned above, we Fig. 4. Adaptation strategies: (a) compression before training, (b) compression propose dataset restoration as an alternative approach that before inference, and (c) dataset restoration. alleviates covariate shift without inducing semantic informa- tion loss in the test data (in contrast to compression before inference). The key idea is to adapt the training dataset using During the process of dataset restoration, we use our trained adversarial image restoration, and use the adapted dataset as generator to restore individually every image in the training actual training data for the downstream task (see Fig. 4c). dataset for the downstream task. Examples of some of the In this way, the on-board analysis module can exploit all the restored images can be found in Figs. 1b and 2. While information available in the captured image. Another important not being able to restore lost semantic information (e.g. the advantage is that adaptation takes place only in the server, and same individual pedestrians), the restored images look sharper, the resulting model can be readily and seamlessly deployed in have fewer artifacts and blurred regions are enhanced with the device with the same hardware, therefore without requiring hallucinated textures that resemble the real images. As such, to install any additional hardware nor increasing the inference the shift with respect to the distribution of original images, on cost. which the trained model is to be evaluated, is reduced. Table I We now recall that a great deal of degradation is related to includes two new configurations OR and RO, where R refers the loss of texture in the decoded image and the appearance of to restored images. compression artifacts (these two factors are clearly apparent in Figs. 1b and 2). Thus, our goal is to find an appropriate B. Adversarial restoration, covariate shift and perceptual in- image restoration technique that could learn from a given set dex of examples and provide us a way to remove the artifacts and recover texture in the images. Perceptual image quality is often assessed using subjective Our restoration module is based on adversarial image evaluations where human subjects are presented with pairs of restoration, where a generative adversarial network (GAN) [1] images where one of them is degraded (generated through conditioned on the degraded image is employed to improve the some artificial processing, such as compression or restoration) image quality. A GAN is based on two networks competing and the other is a real, not-degraded image. The perceptual in an adversarial setting. The generator takes the input image quality is (inversely) proportional to the probability of cor- and outputs the restored image. The discriminator observes rectly selecting the real image. Blau and Michaeli [58] show real and restored images and it is optimized to classify that this probability, and therefore the perceptual quality, can between real and restored images. The generator, in contrast be related to the divergence d(pX , pZ ) (in principle it could is optimized to fool the discriminator, and indirectly improves be any probabilistic divergence) between the distribution of the quality of the restored images. Through the process, the real images pX and the distribution of generated images pZ . generator learns to remove compression artifacts and replace This probabilistic divergence is termed as perceptual index. unrealistic textures by realistic ones that could be used by the The lower the value of the perceptual index, the higher is the discriminator to identify the restored images. The architecture quality of the image. of GAN is based on the one proposed in [56] (for image-to- In practice, collecting human opinions is expensive and image translation [57]), which has a generator and multiple often infeasible. Hence, [58] proposes other practical methods discriminators (see Appendix B for details). to estimate the perceptual index. Specifically, since the task 7 Perceptual index Restoration low distortion, as a lower perceptual index corresponds to a Compressed Restored lower covariate shift. Hence, we use adversarial restoration and decrease the perceptual index at the cost of increased Better perceptual quality distortion. Are all image restoration approaches helpful? We argue that only adversarial image restoration are suitable, since Attainable at r1 and r2 they explicitly minimize the perceptual index through the discriminator and consequently the covariate shift with respect Unattainable at r1 Rate = r1 but attainable at r2 to the captured images. In contrast, non-adversarial image r1 < r2 restoration methods do not necessarily reduce the perceptual Unattainable at r1 and r2 Rate = r2 index. Typically, these methods try to further decrease the distortion and this can be counter-productive as perception and Distortion distortion are at odds near the limit. Raw images Lower distortion Fig. 5. Illustration of the perception-distortion tradeoff [58], [59]. Restoration C. Training data for dataset restoration process shifts the compressed images from a point of low distortion and low Training the GAN for dataset restoration requires original perceptual quality to a point of higher distortion but higher perceptual quality in the perception-distortion plane. images as the discriminator is tasked to distinguish between the image output from the generator and original images. We consider two cases depending on the data available: of a discriminator neural network is precisely to distinguish a. Privileged dataset. We assume the availability of some between real and artificial images, its success rate could be amount of original images from the same distribution used as perceptual index. With this point of view, training (i.e. pX ) that can be used to train the restoration net- a GAN can be seen as decreasing the perceptual index of work (e.g. collected with lossless compression). These generated images by decreasing the perceptual index measured privileged images are generally much more expensive by the discriminator. However, a discriminator needs to be to collect than the usual (lossily) compressed images. trained for every experiment, and also requires many images. Note that for configurations RO and OR, the restoration As a more practical solution, [58] also suggests that Blind network is trained using privileged data. Image Quality Assessment methods can be a suitable proxy for b. Auxiliary dataset. We use an external dataset Z with the perceptual index, since these methods are trained to predict uncompressed images, preferably from a similar domain. the actual human opinion scores in image quality assessment This option has typically zero cost, since we can leverage tests. publicly available image restoration datasets, or even With the interpretation of perceptual index as a divergence directly use a publicly available adversarial restoration with respect the distribution of real images, we observe an model. We denote as AO the configuration RO when the interesting relation between perceptual index and covariate restoration network is trained using an auxiliary dataset. shift, which explains why adversarial image restoration is an The images in Z follow a distribution pZ 6= pX . Thus, appropriate approach (compared to non-adversarial). In the CO training the restoration network with the auxiliary dataset configuration, the training images Xtr are compressed and has the drawback of suffering from certain domain shift, therefore follow pX̂ , while the test images Xts follow pX . which does not occur in the privileged dataset. However, the Thus, the covariate shift can be quantified as d(pX , pX̂ ), where degradations and artifacts that a restoration network restores d denotes a probabilistic divergence. Note that this quantity tend to be local and low-level, which are largely shared is essentially the perceptual index of compressed images. across different domains. In general, dataset restoration with an Therefore, in the case of CO configuration, the covariate shift auxiliary dataset is already effective and a budget option, while in the configuration corresponds to the perceptual index of a privileged dataset without domain shift is more effective, but the training set. Now, an important conclusion from [58] is incurs the additional cost of collecting it. that perception and distortion are at odds with each other, and that there exists a limit beyond which perception and V. E XPERIMENTS distortion cannot be reduced simultaneously (see Fig. 5). Thus, the effect of dataset restoration (i.e. moving from CO to RO) A. Experimental settings is to lower the covariate shift at the expense of increasing Datasets. We evaluate our methods on three datasets: distortion, provided we are close to the perception-distortion Cityscapes [60] is a popular dataset in autonomous driving, limit. and contains 5000 street images (2975/500/1525 for train- We are ultimately interested in the implications on the ing/validation/test sets) of which training and validation have performance of the downstream task, semantic segmentation pixel-level segmentation maps annotated with 19 different in particular. Our analysis in the previous section reveals concepts, including objects and “stuff”. We use the annotated that the task performance is greatly dependent on covariate sets to train (training set) and evaluate semantic segmentation shift and semantic information loss. Reducing the perceptual (validation set). It also contains another 20000 images with index is therefore more crucial than training with images of coarse annotation. We ignore these annotations and use a 8 subset of 2000 images to train the deep image compression model (i.e. MSH [10]) and the image restoration methods. INRIA Aerial Images Dataset [61] contains aerial images of diverse urban settlements with segmentation maps with two classes (building and background). The dataset consists of aerial images from 10 cities with 36 images per city. Annota- tions are provided for 5 of these cities and the segmentation models were trained on 4 cities and evaluated on 1 from these. The images from the other 5 cities were used for compression and restoration. Semantic Drone Dataset [62] contains 400 high resolution images captured with an autonomous drone at an altitude of 5 to 30 meters above ground, and their corresponding annotated segmentation maps (20 classes). The 400 publicly released images were resized from a resolution of 6000x4000 to 3000x2000. The segmentation models were trained on 265 Fig. 6. Rate-distortion curves for Cityscapes for BPG and MSH, and with and without adversarial image restoration. images and evaluated on 70 images while the remaining 65 images were used for the compression and restoration models. Each image was further split into 12 patches each 80 with dimension of a 1200x800. All metrics are calculated on these patched images. Compression methods. We use two state-of-the-art image compression methods. The Better Portable Graphics (BPG) format [7] is based on a subset of the video compression stan- 70 dard HEVC/H.265 [8] and is the state-of-the-art in non-deep mIoU (%) image compression. The Mean Scale Hyperprior (MSH) [10], [13] is a state-of-the-art deep image compression method, based on an autoencoder whose parameters are learned to 60 jointly minimize rate and distortion at a particular tradeoff λ, i.e. min R + λD. MSH models were pretrained for 600k 50 OO OC - MSH RO - BPG iterations on the CLIC Professional Dataset5 with MSE loss. CO - MSH OR - MSH AO - BPG 40 CC - MSH CO - BPG OC - BPG Appendix A contains details of the model architecture. 30 RO - MSH CC - BPG OR - BPG 20 AO - MSH Restoration methods. Our adversarial restoration architec- 10 0.04 0.06 0.08 0.10 0.12 ture for the proposed dataset restoration method is based on bpp FineNet [63]. FineNet is an adaptation of Pix2PixHD [56], a popular GAN architecture used for a broad range of image- Fig. 7. Segmentation performance on Cityscapes for different training/test configurations. to-image translation problems. Refer to Appendix B for more details. Further, when comparing adversarial and non- adversarial approaches, we use Residual Dense Network [30] (Appendix C) as a representative method of non-adversarial approach on them (see Fig. 6). The curves sweep the whole restoration. range, from low to high quality images. As expected, the dis- Segmentation. For the downstream task we use the state-of- tortion decreases (PSNR increases) with rate. We observe that the-art semantic segmentation method DeepLabv3+ [64]. The MSH performs significantly better than BPG on Cityscapes, model is trained using the same procedure mentioned in the i.e. it produces images with lower average distortion at similar paper. We use an output stride of 16 and perform single scale rate. Due to the perception-distortion tradeoff, restoration leads evaluation. to increase in the average distortion, which can be observed Metrics. The quality of the inferred semantic segmentation in Fig. 6. Interestingly, once the images are restored, images map is evaluated using the mean intersection over union compressed with MSH have marginally higher distortion than (mIoU, the higher the better). For image compression we those compressed with BPG. measure rate in bits per pixel (bpp, the lower the better) and the distortion in PSNR (in dB, the higher the better). Segmentation performance. We evaluate the segmentation performance under seven different configurations (i.e. OO, CO, RO, AO, CC, OC and OR). The results are shown in Fig. 7. B. Cityscapes Rate-distortion curves. We first characterize the rate- For the Cityscapes dataset, we observe that the model with distortion performance for the two compression methods in configuration CO outperforms the model with configuration our experiments, and the impact of the proposed restoration CC, which shows that correcting covariate shift by compres- sion before inference can potentially result in lowering the 5 https://www.compression.cc/2019/challenge/ performance. Table II shows the performance per class. We 9 TABLE II P ER CLASS SEGMENTATION PERFORMANCE FOR THE CONFIGURATIONS OO, CO, RO, AO AND CC. light sign rcycle tation alk ing le in n c c Sidew e Truck Bicyc Build Perso Rider Traffi Traffi Moto Train Road Terra Small Big Mean Vege Fenc Wall Pole B us Sky Car bpp obj. obj. IoU RO (BPG) AO (BPG) CC (BPG) CO (BPG) RO (MSH) AO (MSH) CC (MSH) CO (MSH) 0.0419 97.27 78.29 90.13 28.36 49.32 90.31 51.83 94.33 93.91 76.13 77.56 48.08 77.94 54.00 52.55 70.39 58.80 59.20 72.95 63.69 72.96 69.55 0.0613 97.50 79.98 91.11 32.74 50.31 91.08 57.65 93.98 94.41 77.82 84.76 65.35 79.22 57.76 61.15 73.32 59.46 62.39 73.63 66.70 76.39 72.82 0.0891 97.93 82.79 91.44 34.13 53.66 91.40 58.81 94.63 94.72 78.28 78.73 42.79 80.47 58.48 62.29 75.07 62.28 64.41 76.03 68.43 74.94 72.54 0.1279 97.77 82.15 91.83 40.36 55.53 91.75 59.48 94.67 94.80 80.08 87.80 75.28 81.16 60.36 63.99 75.34 62.49 65.16 77.20 69.39 79.29 75.64 0.0419 97.17 76.68 89.12 50.18 42.21 88.57 51.99 94.31 92.58 73.95 77.13 56.96 71.46 50.95 43.81 64.12 52.25 53.30 66.97 57.55 74.24 68.09 0.0613 97.56 79.99 90.26 51.93 46.02 89.96 59.22 94.53 93.52 77.54 81.73 52.43 74.54 54.46 52.66 68.12 55.70 56.76 70.41 61.81 76.22 70.91 0.0891 97.70 81.17 91.03 48.93 49.88 90.82 58.53 94.83 94.18 78.69 78.60 43.80 76.86 54.80 55.03 71.61 58.81 60.41 73.77 64.47 75.68 71.55 0.1279 97.84 82.24 91.62 53.64 52.42 91.41 59.93 94.75 94.34 77.05 88.14 72.19 78.95 60.26 60.45 73.17 60.27 63.29 75.48 67.41 79.63 75.13 0.0419 97.44 79.04 90.81 44.23 47.51 90.43 56.46 94.52 94.32 81.46 84.22 61.96 76.85 55.17 55.54 70.97 56.35 59.16 71.67 63.67 76.87 72.01 0.0613 97.62 80.62 91.45 51.81 51.82 91.06 57.77 94.61 94.37 76.96 86.44 77.55 77.94 57.34 59.61 73.13 59.39 61.50 74.38 66.18 79.34 74.49 0.0891 97.57 80.62 91.83 50.42 55.02 91.53 58.70 94.90 94.77 79.81 84.40 63.53 79.35 58.43 63.62 75.04 60.99 63.85 75.62 68.13 78.59 74.74 0.1279 97.42 80.44 92.01 52.22 55.05 91.80 59.92 94.93 95.07 82.16 83.85 63.37 80.23 58.78 63.35 74.88 62.23 65.63 76.60 68.82 79.02 75.26 0.0419 97.67 80.85 91.40 51.17 49.80 91.07 59.97 94.87 94.30 80.74 86.69 70.06 77.95 57.76 61.35 72.64 58.87 61.37 73.47 66.20 79.05 74.32 0.0613 97.97 83.22 91.90 54.29 53.18 91.55 61.23 94.86 94.45 76.28 82.31 68.31 78.98 57.71 62.51 74.65 60.96 63.92 75.60 67.76 79.13 74.94 0.0891 97.77 82.23 92.28 55.29 57.61 92.04 61.96 94.91 94.99 81.59 86.45 69.66 80.28 60.05 63.24 75.57 61.83 65.48 76.75 69.03 80.57 76.31 0.1279 97.77 82.38 92.43 53.87 56.75 92.23 62.40 95.00 95.09 79.92 88.71 74.90 80.84 60.46 66.08 76.59 63.57 67.00 77.61 70.31 80.96 77.03 0.0454 97.49 79.67 90.34 35.84 48.22 90.73 55.80 94.03 93.87 75.86 81.80 60.63 77.44 55.50 61.08 71.01 56.55 59.69 72.75 64.86 75.36 71.49 0.0674 97.72 81.54 91.27 41.55 51.88 91.23 59.61 94.77 94.36 75.77 82.64 59.23 79.19 58.28 58.26 72.98 59.39 62.70 75.16 66.56 76.80 73.03 0.0870 97.98 83.15 91.33 44.33 52.73 91.42 58.81 94.90 94.46 75.11 81.91 58.77 79.82 58.48 62.55 74.21 59.75 62.04 75.46 67.47 77.08 73.54 0.1279 97.87 83.10 92.06 51.82 56.04 91.94 60.49 95.11 94.62 80.75 87.25 75.90 80.60 60.60 63.86 74.68 62.29 65.56 76.95 69.22 80.58 76.39 0.0454 97.03 76.14 88.66 41.33 42.06 88.48 53.96 94.08 92.54 72.33 76.42 55.60 70.82 50.47 48.80 63.93 51.06 53.41 66.92 57.92 73.22 67.58 0.0674 97.46 79.06 90.15 47.74 47.00 89.68 57.99 94.42 93.27 72.14 79.84 59.16 74.33 53.64 48.00 67.46 55.30 57.01 70.92 60.95 75.66 70.24 0.0870 97.58 80.22 90.50 47.41 47.20 90.25 58.40 94.54 93.58 75.20 83.35 57.80 75.77 54.76 52.58 69.71 56.64 57.73 72.09 62.75 76.34 71.33 0.1279 97.71 81.28 91.35 49.49 52.74 91.10 58.25 94.82 94.16 73.89 88.41 70.80 78.11 58.23 58.27 72.40 59.67 61.75 74.68 66.16 78.67 74.06 0.0454 97.51 79.73 91.04 45.38 48.27 90.81 57.10 94.55 94.36 81.44 85.13 74.56 77.34 56.23 58.89 72.52 57.51 61.33 72.30 65.16 78.25 73.47 0.0674 97.67 81.13 91.86 54.33 54.61 91.40 57.30 94.82 94.46 82.53 84.56 63.39 78.86 57.97 58.46 73.55 60.17 63.75 74.09 66.69 78.99 74.47 0.0870 97.42 80.18 91.89 53.25 51.30 91.65 59.39 94.65 94.64 78.62 85.76 69.54 79.71 59.47 59.62 74.63 61.01 63.92 75.82 67.74 79.02 74.87 0.1279 97.70 81.86 92.04 48.35 52.93 91.94 62.68 95.03 94.91 81.08 88.30 71.81 80.72 60.90 64.78 75.66 62.09 65.61 77.13 69.56 79.89 76.08 0.0454 97.78 81.69 91.52 50.05 52.45 91.24 56.78 94.97 94.18 78.88 86.13 73.22 77.88 56.31 62.19 73.15 59.39 62.81 74.47 66.60 79.07 74.48 0.0674 97.95 82.89 91.84 48.17 52.14 91.57 59.47 94.85 94.53 78.86 85.97 76.59 79.37 58.51 62.00 74.07 60.90 65.14 75.25 67.89 79.57 75.27 0.0870 97.71 81.73 92.13 52.62 54.76 91.75 56.93 94.82 94.81 83.52 88.36 78.94 80.10 59.45 62.72 74.63 62.07 64.93 76.44 68.62 80.67 76.23 0.1279 97.58 81.57 92.54 53.23 58.71 92.05 61.16 94.85 95.05 79.79 85.14 69.27 80.91 60.34 64.78 76.11 63.11 66.83 77.84 69.99 80.08 76.36 OO 9.02 98.17 85.04 92.84 54.96 61.15 92.63 64.62 95.05 95.50 85.62 87.02 70.18 82.57 63.08 66.79 78.00 64.95 69.70 78.69 71.97 81.90 78.24 (PNG) see that the mIoU of classes representing small objects6 is with auxiliary data. In the following subsections, however, we significantly lower in the configuration CC when compared consider different training sets for restoration. to CO. Since smaller objects are relatively easy to lose by Restoration network (auxiliary data). As described in compression, the observation confirms that the introduction of Section IV, we train the restoration network with an auxiliary semantic information loss in the test set from the process of dataset to evaluate the effectiveness of dataset restoration compression before inference is responsible for the decrease when privileged data is not available. We use the images in performance. from the front center camera of the Ford Multi-AV Seasonal The proposed dataset restoration approach, RO is able to Dataset [65] as an auxiliary dataset. improve 1.4-4.8% on the configuration CO, and we achieve Note that, while both being driving datasets, there are close to optimal performance (77% mIoU) requiring only many differences between the Ford dataset and the Cityscapes 0.13 bits per pixel. Thus, lossy compression can result in dataset. Cityscapes is a rich and diverse dataset collected from huge storage savings during data collection when compared to multiple cities in Germany. The images in the Ford dataset are lossless image compression. For the same budget required to obtained by driving a car along a single route in Michigan, collect the 2975 training images with MSH at 0.13 bpp, using USA and hence it lacks diversity. The camera sensors and the lossless compression (PNG in our experiments, resulting in 9 resolution of the images differ as well. bpp) we would have collected only 42 images, which is clearly The configurations AO, RO, CO and CC are compared in insufficient to train the segmentation network. Furthermore, for Fig. 7 and Table II. Despite all the aforementioned differences the same performance, dataset restoration effectively reduces between the datasets, we observe only a small decrease in the required budget (for example, configuration CO achieves terms of performance when AO is compared to RO. The 75.64% with 0.128 bpp, while RO approximately requires configuration AO still performs significantly better than the around 0.07 bpp). baselines, CO and CC. Note that the results for AO depend So far in this section, we have considered the availability on the auxiliary dataset collected which could be improved of privileged data. This allowed us to avoid any domain with a better auxiliary dataset. shift effects that arise when restoration network is trained Amount of privileged data. Since collecting privileged 6 We consider classes person, rider, motorcycle, bicycle, pole, traffic light, data is expensive, the amount of collected images is an traffic sign as small objects and the remaining classes as big objects. important factor. This privileged data is readily available in the 10 TABLE III S EGMENTATION PERFORMANCE OF HYBRID TRAINING SETS COMPARED AGAINST OTHER CONFIGURATIONS . Compression method CO rO o+c O o+r O MSH at 0.0419 bpp 69.55 74.07 75.96 73.99 MSH at 0.0891 bpp 72.54 75.69 77.08 75.88 BPG at 0.0454 bpp 71.49 72.32 74.42 73.98 BPG at 0.0674 bpp 73.03 74.01 75.87 76.12 BPG at 0.0870 bpp 73.54 75.15 75.98 76.67 The original images form 12.5% of all mixtures and the configuration oO results in a performance of 71.90. The IoU of the Train class affects the mIoU dramatically affects BPG at 0.0454 bpp. When Train class is excluded from mIoU, the values for o+c O and o+r O become 74.59 and 74.96. Hybrid training sets. Since privileged and compressed images are available in the server, we now consider hybrid training sets for the segmentation network where 12.5% of Fig. 8. Performance of models with different configurations obtained by the images (373 images) are privileged and the remaining varying the amount of privileged data. are compressed or restored, and evaluate on original images (configuration o+c O or o+r O, respectively). Since we intend to evaluate the segmentation network on original images, we server side, and could be leveraged to train the segmentator, emphasize the contribution of the original images in the loss restoration network or both. In the following experiments, we during training in order to achieve a higher performance. For consider three different amounts of privileged data, 12.5% all the experiments reported in Table III, the loss from original (373 images), 25% (745 images) and 50% (1489 images) of images and compressed (or restored) images are weighed in the size of the segmentation dataset (2975 images). the ratio of 5:1 (empirically determined). First, we evaluate the segmentation network trained solely We observe that the models trained with hybrid training sets with different amounts of original images from the privileged perform better than the individual components of the mixture; dataset. These configurations are oO - 12.5%, oO - 25% and i.e., configuration o+c O performs better than CO and oO, oO - 50%. while o+r O performs better than oO and rO. Between the Next, we train the restoration network with the three dif- models of the two mixtures, we see that dataset restoration can ferent amounts of privileged data mentioned above. The seg- still help performance in the case of BPG. For MSH, dataset mentation network is then trained on the images obtained after restoration may not be necessary. restoring the compressed data using the respective restoration Segmentation performance vs. data collection cost. This models. These configurations are rO - 12.5% (373 images), work is motivated by the need to reduce the data collection rO - 25% (745 images) and RO (1489 images)7 . cost. However, privileged, compressed and auxiliary images Fig. 8 shows the results for the various configurations have different costs associated with their collection (high, mentioned above. For all the different amounts of privileged medium-low, and zero, respectively). In order to provide a data considered, dataset restoration is able to improve the complete picture, in Fig. 9, we plot the performance of all performance over CO. When the privileged data collected is the configurations considered thus far against the total cost 12.5% of the original images, the original images themselves involved in collecting the required data (for training restoration are insufficient to provide a good performance as the config- and segmentation networks). The cost is reported in terms of uration rO - 12.5% performs better than oO - 12.5% at all the percentage of the total cost for the OO configuration. We rates. When 25% privileged data is available, the picture is a see that in the low cost region (≤ 1%), dataset restoration with bit different. At low bitrates, training the segmentation network auxiliary data provides the best performance, since auxiliary with privileged data is sufficient, while at higher bitrates data involves no cost. When a higher performance is needed, restoration is beneficial. At rates greater than 0.087 bpp, rO - privileged data needs to be collected and thus a higher cost is 25% performs better than oO - 25%. However, when privileged incurred. In such cases, the cost due to the rate of compression data is available in copious amounts (e.g. 50%), restoration is dominated by the cost in collecting the privileged data. performs worse even at high bitrates and privileged data can Hybrid training sets, particularly o+c O with MSH and o+r be directly used for training the segmentation network. Thus, O with BPG, result in the best performance in the high cost to understand the benefits of restoration better, we consider region. When the budget is very high (e.g. 50% or higher), training the segmentation network with both restored and lossless compression can be directly used to collect images, original images, and then compare the data collection cost although our motivation in the first place is precisely to avoid against segmentation performance for all the configurations in those very high budget requirements. the following subsections. Adversarial vs. non-adversarial restoration. In order to show the connection between perceptual index and adversarial 7 The configuration RO is the same as rO - 50%. image restoration, we compute the perceptual index of the 11 Fig. 9. Segmentation performance of each configuration against cost of collecting data. Marker size indicates rate. TABLE IV P ERCEPTUAL INDICES OF DIFFERENT IMAGE SETS . TABLE V C OMPARISON OF ADVERSARIAL RESTORATION AGAINST RDN IN Restoration type VARIOUS METRICS . Non Adversarial Compression method Original Compressed Adversarial RDN - P RDN - M Restoration type Compression Evaluation MSH at 0.0419 bpp 40.21 55.69 40.56 55.40 55.82 None Non adversarial Adversarial method metric BPG at 0.0674 bpp 40.21 52.95 41.10 53.69 53.62 (CO) RDN - P RDN - M (RO) mIoU 69.55 71.63 71.82 74.32 RDN - P and RDN - M stand for RDN (PSNR) and RDN (MS-SSIM) MSH at PSNR (dB) 33.47 33.67 33.51 31.55 respectively. The perceptual indices are calculated using the Blind image 0.0419 bpp MS-SSIM (dB) 13.43 13.66 13.66 11.87 quality assessment method of HOSA [66]. mIoU 72.82 72.39 71.92 74.94 MSH at PSNR (dB) 35.00 35.17 35.00 32.86 0.0613 bpp MS-SSIM (dB) 14.90 15.11 15.10 13.08 mIoU 71.49 70.91 69.58 74.48 BPG at PSNR (dB) 33.37 34.04 33.62 31.95 0.0454 bpp MS-SSIM (dB) 12.96 13.58 13.57 11.74 mIoU 73.03 73.26 73.96 75.27 BPG at PSNR (dB) 34.81 35.56 35.19 33.40 0.0674 bpp MS-SSIM (dB) 14.31 14.99 14.96 13.01 RDN - P and RDN - M stand for RDN (PSNR) and RDN (MS-SSIM). Fig. 10. Adversarial dataset restoration vs non-adversarial: While Table IV describes the average perceptual index of the original, compressed and restored image sets, this figure shows the distribution of the perceptual index. Dataset restoration with adversarial image restoration can recover the distribution of the perceptual indices of the original images, while non-adversarial cannot. original, compressed and restored images using the blind quality assessment method HOSA [66] (as described in [58]). We use Residual Dense Networks [30] as a representative method of non-adversarial image restoration. RDN is trained with the objective of increasing the PSNR (RDN - PSNR) or MS-SSIM [67] (RDN - MS-SSIM). Refer to Appendix C for more details. Note that adversarial restoration is able to achieve a perceptual index close to that of the original images Fig. 11. Feature correlation between segmentation features (at a shallow layer) while RDN does not affect the perceptual index significantly. of the segmentation model trained on raw images. Dataset restoration with While the previous experiment shows that adversarial adversarial image restoration can recover the distribution of perceptual index restoration does improve the perceptual quality of images, and segmentation features of the optimal case (OO), while non-adversarial cannot. we are ultimately interested in the segmentation task. We 12 TABLE VI I NFERENCE TIMES FOR DIFFERENT APPROACHES . 80 Compression Segment Encode + Decode + Segment method OO / CO / RO CC MSH 0.64s 0.10s + 0.13s + 0.64s = 0.87s BPG 0.64s 1.10s + 0.15s + 0.64s = 1.89s mIoU (%) Note that Encode and Decode times for MSH and BPG are measured on a 70 GPU and a CPU respectively. Segmentation time is measured on a GPU. 60 use the model trained with the original images and extract features from a shallow layer. Features are obtained from 50 OO OC - MSH RO - BPG original images, compressed images and restored images for CO - MSH OR - MSH OC - BPG 40 CC - MSH CO - BPG OR - BPG comparison. As compression affects the low-level information RO - MSH CC - BPG 30 in images by degrading the texture, adding blur and other alien 0.05 0.10 0.15 0.20 0.25 0.30 0.35 artifacts, a shallow layer is selected to observe how these low- bpp level differences between the images are reflected in semantic Fig. 12. Segmentation performance on INRIA Aerial Images Dataset for features. Since we use the same model to extract the different different training/test configurations. features, they are all aligned in the channel dimension. We want to measure the level of alignment in activation, so we plot the average activation (over the validation set of Cityscapes) that the use of MSH, which causes smoothing artifacts, de- value for each channel of original, compressed or restored stroys the discriminative features and we are unable to learn versus the original one (see Fig. 11). Obviously, the points a segmentation model capable of taking advantage of these corresponding to original images lie on the identity line, features in the original image. This is especially critical with while many channels from compressed images are clearly not the AID since there are only a few features that discriminate aligned. Adversarial restoration manages to bring back the a building from the background. features to the identity line, while non-adversarial restoration The proposed approach of RO performs consistently better has little effect. This shows that adversarially restored images than both these configurations with gains up to 3.9 % mIoU. are not only perceptually closer to the real images, but also Fig. 13 shows a portion of an image from the dataset along semantically more correlated. with the segmentation maps predicted by various models. Further, Table V compares the performance of different Interestingly, contrary to the results on the Cityscapes restoration methods in the configuration RO. Along with dataset, the performance of configuration OR does not improve segmentation performance in terms of mIoU, distortion mea- over OC. This suggests that the process of restoration causes sures of PSNR and MS-SSIM8 are also reported. Adversarial further damage (over compression) to the discriminative fea- restoration results in a far better segmentation performance tures used by the model trained on the original images for its when compared to RDN. prediction. Efficiency. Table VI reports the inference times for different configurations. The time for encode-decode times for MSH D. Semantic Drone Dataset and the segmentation time for DeepLabv3+ were measured Segmentation performance. The segmentation results ob- on a Quadro RTX6000 GPU, while the encode-decode times tained on the Semantic Drones Dataset (SDD) are shown in for BPG was measured on a Intel Xeon(R) E5-1620 v4 CPU9 . Fig. 14. We observe that the models with configuration CC Inference using dataset restoration (configuration RO) is faster outperform CO consistently for both MSH and BPG. We than compression before inference (configuration CC) by 26% attribute this result to the lack of semantic information loss in and 66%, when the compression method used is MSH and this dataset. A typical object from the SDD covers a significant BPG respectively. portion of the image and compression does not significantly affect its recognizability. When we assume little to no semantic C. INRIA Aerial Images Dataset information loss, the effects of covariate shift are dominant Segmentation performance. Fig. 12 depicts the segmen- and as such, the configuration CC performs better. This result tation results obtained on the INRIA Aerial Images Dataset shows that the performance of various configurations are (AID) for the same six configurations mentioned in Section dependent on the properties of the dataset. B. We observe that the model with configuration CO performs The proposed approach of RO performs similarly to config- better than CC when the compression method used is BPG. uration CC and lies within ±1.5% mIoU of the configuration However, the same cannot be said for MSH. We hypothesize CC. The lack of significant semantic information loss in SDD affects the effectiveness of dataset restoration. 8 MS-SSIM (dB) is calculated from the standard MS-SSIM value (range [0, 1]) as follows: MS-SSIM (dB) = −10 log10 (1− MS-SSIM). MS-SSIM is VI. C ONCLUSIONS presented in terms of this logarithmic scale for better distinction. 9 We used the software implementation of BPG from The rapid development in sensor quality and increasing data https://github.com/mirrorer/libbpg using the default options. collection rate makes lossy compression necessary to reduce 13 Fig. 13. Top: (from left to right) - captured image, compressed (BPG), restored (BPG), compressed (MSH), restored (MSH). Bottom: Prediction map and accuracy score of segmentation models with different configurations. Conv: 5 x 5 x 192 / ↓ 2 Conv: 5 x 5 x 192 / ↓ 2 Conv: 5 x 5 x 192 / ↓ 2 Conv: 5 x 5 x 192 / ↓ 2 Conv: 5 x 5 x 192 / ↓ 2 Conv: 5 x 5 x 192 / ↓ 2 75 Conv: 3 x 3 x 192 Raw Image ReLU ReLU GDN GDN GDN Q Q 70 Range Range Encoder Encoder Fully Factorized 11000011 11000011 Entropy , 01011001 01011001 Model 01001101 01001101 65 Conv: 5 x 5 x 192 / ↑ 2 Conv: 5 x 5 x 192 / ↑ 2 Conv: 5 x 5 x 192 / ↑ 2 Conv: 5 x 5 x 192 / ↑ 2 Conv: 5 x 5 x 192 / ↑ 2 mIoU (%) Conv: 5 x 5 x 3 / ↑ 2 Conv: 3 x 3 x 384 Reconstructed Range Range Decoder Decoder Image ReLU ReLU IGDN IGDN IGDN 60 55 Fig. 15. MSH Architecture: Refer to [13] for details on Fully Factorized 50 OO OC - MSH RO - BPG Entropy Model and Range Encoder-Decoder. CO - MSH OR - MSH OC - BPG 45 CC - MSH CO - BPG OR - BPG RO - MSH CC - BPG 40 A PPENDIX B 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45 0.50 bpp A DVERSARIAL IMAGE RESTORATION Fig. 14. Segmentation performance on Semantic Drones Dataset for different We use the FineNet from Akbari et. al [63] (adapted training/test configurations. from [56]) with slight changes for our image restoration module. Following the same notation in [63], the generator ar- transmission and storage costs. By means of dataset restora- chitecture is written as c64 , d128 , d256 , d512 , 9 × tion, we enable the incorporation of lossy compression for r512 , u256 , u128 , u64 , o3 where on-board analysis, greatly mitigating the drop in performance. • ck : Conv: 7 × 7 x k, Instance Normalization, ReLU Moreover, dataset restoration is a principled approach, based • dk : Conv: 3 x 3 x k / ↓ 2, Instance Normalization, ReLU on our analysis of the various scenarios involving learning and • rk : Conv: 3 x 3 x k, Reflection padding, Instance Nor- inference with compressed images. This analysis framework malization, ReLU involving covariate shift and semantic information loss can be • uk : Conv: 3 x 3 x k / ↑ 2, Instance Normalization, ReLU further extended to other degradations like blur, noise, color • o3 : Conv: 7 x 7 x 3, Instance Normalization, Tanh and illumination changes, etc. We use two discriminators, as in [63], operating at two different scales. Akbari et al. rescale the image to half the resolution while we do not. The discriminators act on the A PPENDIX A original resolution, H × W and H/4 × W/4 resolution. M EAN SCALE HYPERPRIOR Again following notation in [63], the discriminators have the following architecture, C64 , C128 , C256 , C512 , O1 , where Fig. 15 describes the architecture of the MSH [13] used in • Ck : Conv: 4 x 4 x k / ↓ 2, Instance Normalization, this paper. LeakyReLU 14 • O1 : Conv: 1 x 1 x 1 ACKNOWLEDGMENT Let the captured image be x. The restored image, x̄ is The authors thank Audi Electronics Venture GmbH, the obtained by adding the residual computed by the generator Generalitat de Catalunya CERCA Program and its AC- to the compressed image, CIO agency for supporting this work. Luis acknowledges x̄ = x̂ + G(x̂). the support of the Spanish project RTI2018-102285-A- I00 and the Ramón y Cajal fellowship RYC2019-027020- All images are scaled to [−1, 1]. I. Joost acknowledges the support of the Spanish project The loss function used for training are as follows: PID2019-104174GB-I00. Antonio acknowledges the support • Generator, G: of project TIN2017-88709-R (MINECO/AEI/FEDER, UE) (G) LGAN + 10 · (2 · L1 + LV GG + LM S−SSIM + LDIST ) and the ICREA Academia programme. i (D ) • Discriminator, Di : LGAN (G) R EFERENCES LGAN is the sum of the standard GAN loss from each of the discriminator, i.e. [1] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in 2 (G) X Advances in neural information processing systems, 2014, pp. 2672– LGAN = − log(Di (x̂, x̄)). 2680. i=1 [2] C. E. Shannon, “A mathematical theory of communication,” Bell system technical journal, vol. 27, no. 3, pp. 379–423, 1948. L1 = kx̄ − xk1 . [3] T. M. Cover and J. A. Thomas, Rate distortion theory, 2nd ed. Elements of information Theory, 1991. LM S−SSIM = MS-SSIM(x̄, x). [4] G. K. Wallace, “The jpeg still picture compression standard,” IEEE transactions on consumer electronics, vol. 38, no. 1, pp. xviii–xxxiv, Let VGG denote a VGG-Net trained on the ImageNet 1992. [5] A. Skodras, C. Christopoulos, and T. Ebrahimi, “The jpeg 2000 still dataset and Mj denote the size of the output of the j th layer image compression standard,” IEEE Signal processing magazine, vol. 18, of VGG. The output of each of the 5 convolution blocks are no. 5, pp. 36–58, 2001. considered for the VGG feature distillation loss, which is given [6] D. Taubman and M. Marcellin, JPEG2000 image compression fun- damentals, standards and practice: image compression fundamentals, by standards and practice. Springer Science & Business Media, 2012, vol. 642. 5 [7] F. Bellard, BPG image format (http://bellard.org/bpg/), 2017. X 1 [8] G. J. Sullivan, J.-R. Ohm, W.-J. Han, and T. Wiegand, “Overview of LV GG = V GG(j) (x̄) − V GG(j) (x) . the high efficiency video coding (hevc) standard,” IEEE Transactions j=1 M j 1 on circuits and systems for video technology, vol. 22, no. 12, pp. 1649– 1668, 2012. Similarly, the features of the discriminators are also distilled [9] J. Ballé, V. Laparra, and E. P. Simoncelli, “End-to-end optimized image for stable GAN training. compression,” arXiv preprint arXiv:1611.01704, 2016. [10] J. Ballé, D. Minnen, S. Singh, S. J. Hwang, and N. Johnston, “Vari- ational image compression with a scale hyperprior,” arXiv preprint 2 X 4 X 1 (j) (j) arXiv:1802.01436, 2018. LDIST = (i) Di (x̂, x̄) − Di (x̂, x) . [11] G. Toderici, S. M. O’Malley, S. J. Hwang, D. Vincent, D. Minnen, Nj 1 S. Baluja, M. Covell, and R. Sukthankar, “Variable rate image compres- i=1 j=1 sion with recurrent neural networks,” arXiv preprint arXiv:1511.06085, The discriminators are trained using the standard GAN loss. 2015. [12] L. Theis, W. Shi, A. Cunningham, and F. Huszár, “Lossy image compres- (D ) i sion with compressive autoencoders,” arXiv preprint arXiv:1703.00395, LGAN = log(1 − Di (x̂, x̄)) + log(Di (x̂, x)). 2017. [13] D. Minnen, J. Ballé, and G. D. Toderici, “Joint autoregressive and We use a batchsize of 1 and train the GAN for around hierarchical priors for learned image compression,” in Advances in 135k iterations. Adam optimizer with β1 = 0.1 and β2 = 0.9 Neural Information Processing Systems, 2018, pp. 10 771–10 780. is employed. Initially, the learning rate is set to 0.0002 and is [14] F. Yang, L. Herranz, J. van de Weijer, J. A. I. Guitián, A. M. López, and M. G. Mozerov, “Variable rate deep image compression with modulated reduced by a factor of 10 after 80k iterations. autoencoder,” IEEE Signal Processing Letters, vol. 27, pp. 331–335, 2020. A PPENDIX C [15] S. Dodge and L. Karam, “Understanding how image quality affects deep neural networks,” in 2016 eighth international conference on quality of R ESIDUAL DENSE NETWORK multimedia experience (QoMEX). IEEE, 2016, pp. 1–6. We use the RDN architecture from [30]. We ask the reader [16] D. Hendrycks and T. Dietterich, “Benchmarking neural network ro- bustness to common corruptions and perturbations,” arXiv preprint to refer to the paper for the architecture. The following arXiv:1903.12261, 2019. hyperparameters are used: Global layers = 16, Local layers [17] P. Roy, S. Ghosh, S. Bhattacharya, and U. Pal, “Effects of degradations = 6, Growth rate = 32. on deep neural network architectures,” arXiv preprint arXiv:1807.10108, We train the CAR model with the objective of maximising 2018. [18] T. S. Borkar and L. J. Karam, “Deepcorrect: Correcting dnn models MS-SSIM or PSNR. The models are trained using 256x256 against image distortions,” IEEE Transactions on Image Processing, patches. A mini-batchsize of 1 is used and the model is trained vol. 28, no. 12, pp. 6022–6034, 2019. for around 200k iterations. Adam optimizer is used with initial [19] S. Ghosh, R. Shet, P. Amon, A. Hutter, and A. Kaup, “Robustness of deep convolutional neural networks for image degradations,” in learning rate of 0.001 which is reduced by a factor 10 at 80k 2018 IEEE International Conference on Acoustics, Speech and Signal and 150k iterations. Processing (ICASSP). IEEE, 2018, pp. 2916–2920. 15 [20] R. Torfason, F. Mentzer, E. Agustsson, M. Tschannen, R. Timofte, and [44] Y. Ganin and V. Lempitsky, “Unsupervised domain adaptation by L. Van Gool, “Towards image understanding from deep compression backpropagation,” arXiv preprint arXiv:1409.7495, 2014. without decoding,” arXiv preprint arXiv:1803.06131, 2018. [45] K. Bousmalis, N. Silberman, D. Dohan, D. Erhan, and D. Krishnan, [21] M. Ehrlich and L. S. Davis, “Deep residual learning in the jpeg transform “Unsupervised pixel-level domain adaptation with generative adversarial domain,” in Proceedings of the IEEE International Conference on networks,” in Proceedings of the IEEE conference on computer vision Computer Vision, 2019, pp. 3484–3493. and pattern recognition, 2017, pp. 3722–3731. [22] J. Löhdefink, A. Bär, N. M. Schmidt, F. Hüger, P. Schlicht, and [46] K. Saito, D. Kim, S. Sclaroff, T. Darrell, and K. Saenko, “Semi- T. Fingscheidt, “Gan-vs. jpeg2000 image compression for distributed supervised domain adaptation via minimax entropy,” in Proceedings of automotive perception: Higher peak snr does not mean better semantic the IEEE International Conference on Computer Vision, 2019, pp. 8050– segmentation,” arXiv preprint arXiv:1902.04311, 2019. 8058. [23] E. Agustsson, M. Tschannen, F. Mentzer, R. Timofte, and L. V. Gool, [47] M. Chen, Z. Xu, K. Weinberger, and F. Sha, “Marginalized denoising “Generative adversarial networks for extreme learned image compres- autoencoders for domain adaptation,” arXiv preprint arXiv:1206.4683, sion,” in Proceedings of the IEEE International Conference on Computer 2012. Vision, 2019, pp. 221–231. [48] X. Glorot, A. Bordes, and Y. Bengio, “Domain adaptation for large- [24] A. Buades, B. Coll, and J.-M. Morel, “A review of image denoising scale sentiment classification: A deep learning approach,” in Proceedings algorithms, with a new one,” Multiscale Modeling & Simulation, vol. 4, of the 28th International Conference on International Conference on no. 2, pp. 490–530, 2005. Machine Learning, ser. ICML’11. Madison, WI, USA: Omnipress, [25] W. H. Richardson, “Bayesian-based iterative method of image restora- 2011, p. 513–520. tion,” JoSA, vol. 62, no. 1, pp. 55–59, 1972. [49] E. Tzeng, J. Hoffman, K. Saenko, and T. Darrell, “Adversarial discrim- [26] L. B. Lucy, “An iterative technique for the rectification of observed inative domain adaptation,” in Proceedings of the IEEE Conference on distributions,” The astronomical journal, vol. 79, p. 745, 1974. Computer Vision and Pattern Recognition, 2017, pp. 7167–7176. [27] S. C. Park, M. K. Park, and M. G. Kang, “Super-resolution image [50] Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Lavio- reconstruction: a technical overview,” IEEE signal processing magazine, lette, M. Marchand, and V. Lempitsky, “Domain-adversarial training of vol. 20, no. 3, pp. 21–36, 2003. neural networks,” The Journal of Machine Learning Research, vol. 17, [28] M.-Y. Shen and C.-C. J. Kuo, “Review of postprocessing techniques no. 1, pp. 2096–2030, 2016. for compression artifact removal,” Journal of visual communication and [51] J. Hoffman, E. Tzeng, T. Park, J.-Y. Zhu, P. Isola, K. Saenko, A. A. image representation, vol. 9, no. 1, pp. 2–14, 1998. Efros, and T. Darrell, “Cycada: Cycle-consistent adversarial domain [29] Y. Chen and T. Pock, “Trainable nonlinear reaction diffusion: A flexible adaptation,” arXiv preprint arXiv:1711.03213, 2017. framework for fast and effective image restoration,” IEEE Transactions [52] Z. Murez, S. Kolouri, D. Kriegman, R. Ramamoorthi, and K. Kim, on Pattern Analysis and Machine Intelligence, 2016. “Image to image translation for domain adaptation,” in Proceedings [30] Y. Zhang, Y. Tian, Y. Kong, B. Zhong, and Y. Fu, “Residual dense of the IEEE Conference on Computer Vision and Pattern Recognition, network for image restoration,” arXiv preprint arXiv:1812.10477, 2018. 2018, pp. 4500–4509. [53] B. Hayes, “Cloud computing,” Communications of the ACM, vol. 51, [31] S. Nah, T. Hyun Kim, and K. Mu Lee, “Deep multi-scale convolutional no. 7, pp. 9–11, 2008. neural network for dynamic scene deblurring,” in Proceedings of the [54] W. Zhu, C. Luo, J. Wang, and S. Li, “Multimedia cloud computing,” IEEE Conference on Computer Vision and Pattern Recognition, 2017, IEEE Signal Processing Magazine, vol. 28, no. 3, pp. 59–69, 2011. pp. 3883–3891. [55] N. Fernando, S. W. Loke, and W. Rahayu, “Mobile cloud computing: [32] L. Chen, F. Fang, T. Wang, and G. Zhang, “Blind image deblurring with A survey,” Future generation computer systems, vol. 29, no. 1, 2013. local maximum gradient prior,” in Proceedings of the IEEE Conference [56] T.-C. Wang, M.-Y. Liu, J.-Y. Zhu, A. Tao, J. Kautz, and B. Catanzaro, on Computer Vision and Pattern Recognition, 2019, pp. 1742–1750. “High-resolution image synthesis and semantic manipulation with con- [33] W. Shi, J. Caballero, F. Huszár, J. Totz, A. P. Aitken, R. Bishop, ditional gans,” in CVPR, 2018, pp. 8798–8807. D. Rueckert, and Z. Wang, “Real-time single image and video super- [57] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation resolution using an efficient sub-pixel convolutional neural network,” in with conditional adversarial networks,” in CVPR, 2017, pp. 1125–1134. Proceedings of the IEEE conference on Computer Vision and Pattern [58] Y. Blau and T. Michaeli, “The perception-distortion tradeoff,” in Pro- Recognition, 2016, pp. 1874–1883. ceedings of the IEEE Conference on Computer Vision and Pattern [34] Y. Zhang, Y. Tian, Y. Kong, B. Zhong, and Y. Fu, “Residual dense Recognition, 2018, pp. 6228–6237. network for image super-resolution,” in Proceedings of the IEEE Con- [59] ——, “Rethinking lossy compression: The rate-distortion-perception ference on Computer Vision and Pattern Recognition, 2018. tradeoff,” arXiv preprint arXiv:1901.07821, 2019. [35] K. Zhang, W. Zuo, Y. Chen, D. Meng, and L. Zhang, “Beyond a gaussian [60] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benen- denoiser: Residual learning of deep cnn for image denoising,” IEEE son, U. Franke, S. Roth, and B. Schiele, “The cityscapes dataset for Transactions on Image Processing, vol. 26, no. 7, pp. 3142–3155, 2017. semantic urban scene understanding,” in Proc. of the IEEE Conference [36] K. Zhang, W. Zuo, and L. Zhang, “Ffdnet: Toward a fast and flexible on Computer Vision and Pattern Recognition (CVPR), 2016. solution for cnn-based image denoising,” IEEE Transactions on Image [61] E. Maggiori, Y. Tarabalka, G. Charpiat, and P. Alliez, “Can semantic Processing, vol. 27, no. 9, pp. 4608–4622, 2018. labeling methods generalize to any city? the inria aerial image labeling [37] C. Dong, Y. Deng, C. Change Loy, and X. Tang, “Compression artifacts benchmark,” in IEEE International Geoscience and Remote Sensing reduction by a deep convolutional network,” in Proceedings of the IEEE Symposium (IGARSS). IEEE, 2017. International Conference on Computer Vision, 2015, pp. 576–584. [62] C. Mostegel, M. Maurer, N. Heran, J. P. Puerta, , and F. Fraundorfer. [38] C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham, A. Acosta, (2019) Semantic drone dataset. [Online]. Available: http://dronedataset. A. Aitken, A. Tejani, J. Totz, Z. Wang et al., “Photo-realistic single icg.tugraz.at image super-resolution using a generative adversarial network,” in [63] M. Akbari, J. Liang, and J. Han, “Dsslic: deep semantic segmentation- Proceedings of the IEEE conference on computer vision and pattern based layered image compression,” in ICASSP 2019-2019 IEEE In- recognition, 2017, pp. 4681–4690. ternational Conference on Acoustics, Speech and Signal Processing [39] X. Wang, K. Yu, S. Wu, J. Gu, Y. Liu, C. Dong, Y. Qiao, and (ICASSP). IEEE, 2019, pp. 2042–2046. C. Change Loy, “Esrgan: Enhanced super-resolution generative adversar- [64] L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam, “Encoder- ial networks,” in Proceedings of the European Conference on Computer decoder with atrous separable convolution for semantic image segmen- Vision (ECCV), 2018. tation,” in Proceedings of the European conference on computer vision [40] L. Galteri, L. Seidenari, M. Bertini, and A. Del Bimbo, “Deep universal (ECCV), 2018, pp. 801–818. generative adversarial compression artifact removal,” IEEE Transactions [65] S. Agarwal, A. Vora, G. Pandey, W. Williams, H. Kourous, on Multimedia, 2019. and J. McBride, “Ford multi-av seasonal dataset,” arXiv preprint [41] Z. Zhao, Q. Sun, H. Yang, H. Qiao, Z. Wang, and D. O. Wu, “Compres- arXiv:2003.07969, 2020. sion artifacts reduction by improved generative adversarial networks,” [66] J. Xu, P. Ye, Q. Li, H. Du, Y. Liu, and D. Doermann, “Blind image EURASIP Journal on Image and Video Processing, 2019. quality assessment based on high order statistics aggregation,” IEEE [42] O. Kupyn, V. Budzan, M. Mykhailych, D. Mishkin, and J. Matas, Transactions on Image Processing, vol. 25, no. 9, pp. 4444–4457, 2016. “Deblurgan: Blind motion deblurring using conditional adversarial net- [67] Z. Wang, E. P. Simoncelli, and A. C. Bovik, “Multiscale structural works,” in CVPR, 2018, pp. 8183–8192. similarity for image quality assessment,” in The Thrity-Seventh Asilomar [43] M. Wang and W. Deng, “Deep visual domain adaptation: A survey,” Conference on Signals, Systems & Computers, 2003, vol. 2. IEEE, 2003, Neurocomputing, vol. 312, pp. 135–153, 2018. pp. 1398–1402.