Domain-Adaptive Deep Network Compression
Masana, Marc (Centre de Visió per Computador)
Weijer, Joost van de (Centre de Visió per Computador)
Herranz, Luis (Universitat Autònoma de Barcelona)
Bagdanov, Andrew (Università degli Studi di Firenze)
Alvarez, Jose M. (Toyota Research Institute)

Imprint:	IEEE, 2017
Abstract:	Deep Neural Networks trained on large datasets can be easily transferred to new domains with far fewer labeled examples by a process called fine-tuning. This has the advantage that representations learned in the large source domain can be exploited on smaller target domains. However, networks designed to be optimal for the source task are often prohibitively large for the target task. In this work we address the compression of networks after domain transfer. We focus on compression algorithms based on low-rank matrix decomposition. Existing methods base compression solely on learned network weights and ignore the statistics of network activations. We show that domain transfer leads to large shifts in network activations and that it is desirable to take this into account when compressing. We demonstrate that considering activation statistics when compressing weights leads to a rank-constrained regression problem with a closed-form solution. Because our method takes into account the target domain, it can more optimally remove the redundancy in the weights. Experiments show that our Domain Adaptive Low Rank (DALR) method significantly outperforms existing low-rank compression techniques. With our approach, the fc6 layer of VGG19 can be compressed more than 4x more than using truncated SVD alone - with only a minor or no loss in accuracy. When applied to domain-transferred networks it allows for compression down to only 5-20% of the original number of parameters with only a minor drop in performance.
Grants:	European Commission 665919 Generalitat de Catalunya 2017FIB-00218 Agencia Estatal de Investigación TIN2016-79717-R Ministerio de Economía y Competitividad PCIN2015-251
Note:	Altres ajuts: CERCA Programme/Generalitat de Catalunya; GPU support from Nvidia
Rights:	Aquest material està protegit per drets d'autor i/o drets afins. Podeu utilitzar aquest material en funció del que permet la legislació de drets d'autor i drets afins d'aplicació al vostre cas. Per a d'altres usos heu d'obtenir permís del(s) titular(s) de drets.
Language:	Anglès
Document:	Capítol de llibre ; recerca ; Versió acceptada per publicar
Subject:	Closed form solutions ; Compression algorithms ; Constrained regression ; Domain transfers ; Low rank compression ; Low-rank matrices ; Network activations ; Network compression
Published in:	Proceedings of the IEEE International Conference on Computer Vision, 2017, p. 4299-4307, ISSN 2380-7504, ISBN 978-1-5386-1032-9

DOI: 10.1109/ICCV.2017.460

Postprint 10 p, 647.7 KB

The record appears in these collections:
Books and collections > Book chapters