This is the accepted version of the article: de Cea Dominguez, Carlos; Moure, Juan C.; Bartrina Rapesta, Joan; [et al.]. «GPU-oriented architecture for an end-to-end image». IEEE Access, Vol. 8 (April 2020), p. 68474-68487. DOI 10.1109/ACCESS.2020.2985859 This version is available at https://ddd.uab.cat/record/234854 under the terms of the license Digital Object Identifier GPU-oriented architecture for an end-to-end image/video codec based on JPEG2000 CARLOS DE CEA-DOMINGUEZ1 , JUAN C. MOURE 2 , JOAN BARTRINA-RAPESTA1 , AND FRANCESC AULÍ-LLINÀS, (Senior Member, IEEE)1 1 Department of Information and Communications Engineering, Universitat Autònoma de Barcelona, Spain (phone: +34 935811861; fax: +34 935813443; e-mail: carlos.decea@uab.cat, joan.bartrina@uab.cat, francesc.auli@uab.cat) 2 Department of Computer Architecture and Operating Systems, Universitat Autònoma de Barcelona, Spain (e-mail: juancarlos.moure@uab.es) Corresponding author: Carlos de Cea-Dominguez (e-mail: carlos.decea@uab.cat). This work has been partially supported by the Spanish Ministry of Economy and Competitiveness and the European Regional Development Fund under Grants TIN2017-84553-C2-1-R and RTI2018-095287-B-I00 (MINECO/FEDER, UE), and by the Catalan Government under Grants 2017SGR-463 and 2017SGR-313. Copyright (c) 2020 IEEE. Personal use of this material is permitted. However, permission to use this material for any other purposes must be obtained from the IEEE by sending a request to pubs-permissions@ieee.org. ABSTRACT Modern image and video compression standards employ computationally intensive algo- rithms that provide advanced features to the coding system. Current standards often need to be implemented in hardware or using expensive solutions to meet the real-time requirements of some environments. Contrarily to this trend, this paper proposes an end-to-end codec architecture running on inexpensive Graphics Processing Units (GPUs) that is based on, though not compatible with, the JPEG2000 international standard for image and video compression. When executed in a commodity Nvidia GPU, it achieves real time processing of 12K video. The proposed S/W architecture utilizes four CUDA kernels that minimize memory transfers, use registers instead of shared memory, and employ a double-buffer strategy to optimize the streaming of data. The analysis of throughput indicates that the proposed codec yields results at least 10× superior on average to those achieved with JPEG2000 implementations devised for CPUs, and approximately 4× superior to those achieved with hardwired solutions of the HEVC/H.265 video compression standard. INDEX TERMS Wavelet-based image coding, high-throughput image coding, JPEG2000, GPU, CUDA. I. INTRODUCTION others. To do so, they use algorithms that scan, transform, and code the samples1 of the image multiple times, consuming VER the past decades, the computational complex- O ity of image and video coding systems has increased notably. In the early nineties, the JPEG standard (ISO/IEC significant processing time even when executed in the latest processors. JPEG2000 is a widespread standard in fields that deal with 10918) [1] employed the low-complexity discrete cosine large sets of images and/or videos. Its coding pipeline has transform [2] and Huffman [3] coding. Ten years after, the three main stages [8]. The first reduces the image redundancy JPEG2000 standard (ISO/IEC 15444) [4] introduced more through a color transform (CT) and the DWT. The second computationally demanding algorithms such as the discrete employs bitplane coding together with arithmetic coding to wavelet transform (DWT) [5] and bitplane coding [6]. In reduce the statistical redundancy of wavelet coefficients. The the last years, HEVC/H.265 (ISO/IEC 23008) [7] doubled third reorganizes the data to produce the final codestream. the compression efficiency of previous standards by using The high computational complexity of these stages poses complex techniques that exploit intra- and inter-redundancy of frames. Nowadays, most codecs (including JPEG2000 1 A sample is the basic unit of a digital image, representing a level of and HEVC) provide advanced features such as scalability by brightness in a grayscale or color component (each RGB pixel has three quality, interactive transmission, and error resilience, among samples). VOLUME VOL, YEAR 1 Carlos de Cea et al.: GPU-oriented architecture for an end-to-end image/video codec based on JPEG2000 a challenge to meet the real-time requirements of some GPUs. The proposed algorithm is not compatible with the scenarios. In Digital Cinema, for instance, JPEG2000 needs standard, but it allows parallel coefficient processing within to be implemented in Field-Programmable Gate Arrays to the codeblock. process 2K (i.e., 2048×1024) and 4K (i.e., 4096×2048) Following a similar line, in 2017 the Joint Photographics resolution [9]. In medical and remote sensing applications, Experts Group launched a call for proposals with the aim to dedicated servers and workstations are employed to manage augment the parallelism in the second stage of the coding and store the large quantity of images that are produced pipeline. This new part of JPEG2000 (ISO/IEC 15444-15) daily [10], [11]. This has motivated many works in the adopts the algorithm proposed in [46]. Such algorithm is literature that propose hardware architectures to accelerate devised to mostly benefit from the modern instruction sets particular stages of the JPEG2000 coding pipeline [12]–[22]. like AVX2, NEON, and BMI2 included in new CPUs, though Highly parallel architectures may help to reduce process- it can also be implemented in GPUs [47]. It is about 10× ing time and costs in some environments. Graphics Process- faster than the standard, but it penalizes coding performance ing Units (GPUs) may be ideal due to their high through- in approximately 10%. Also, it sacrifices quality scalability, put, low cost, and widespread availability. Their architecture which is a valued feature of the system since it permits the is mainly based on the Single Instruction Multiple Data transmission of an image progressively by quality. (SIMD) paradigm, which executes a flow of instructions This paper introduces a highly-parallel, GPU-oriented on multiple data in a lock-step synchronous way. When codec based on JPEG2000. The proposed codec is the final the program allows data (in addition to task) parallelism, piece of our research line that was aimed to explore new cod- thousands of threads can be executed in parallel, achieving ing techniques for image/video compression tailored for the a throughput that is potentially an order of magnitude higher fine-grain parallelism of GPUs. The JPEG2000 framework is than that achieved by conventional Central Processing Units employed to show that the proposed techniques can virtually (CPUs) [23]. This is in part because the architecture of the obtain the same coding performance of this standard without CPUs is more based on the Multiple Instruction Multiple sacrificing any feature. Evidently, compliance with the stan- Data (MIMD) paradigm, which allows the asynchronous dard is lost since the proposed techniques require significant execution of fewer threads over different sets of data. changes in the core coding system. A preliminary version Most of the workload in the first stage of the JPEG2000 of the proposed codec was partially described in [48], [49]. pipeline lies in the DWT, which is well-suited to the This paper vastly improves our previous work by describing SIMD paradigm. The first implementations of the DWT for the complete coding pipeline with the needed machinery GPUs appeared in the 2000s making use of the graphics to avoid bottlenecks, providing the color transform and the pipeline [24]–[26]. Later, the use of the Compute Unified codestream reorganization stages with an in-depth analysis Device Architecture (CUDA) programming language intro- of the kernel metrics and memory transfers, and reporting ex- duced by Nvidia increased the throughput of such imple- tensive experimental tests. The obtained results show that the mentations significantly [27]–[31]. Recently, we proposed a proposed S/W architecture can process real-time 12K (i.e., register-based implementation of the DWT for GPUs [32] 12288×6144) video, achieving a throughput 4× superior to that yields 40× speedups compared to CPU implementations. that achieved by the state-of-the-art Nvidia codec of HEVC Similar results are also achieved in [33]. that is supported by in-chip dedicated hardware. In general, the DWT takes 15% of the total execution time The rest of the paper is structured as follows. Section II of the codec. The most expensive stage is the bitplane and briefly overviews the architecture of Nvidia GPUs and arithmetic coding, which spends about 80% of the time. This JPEG2000. Section III describes the proposed codec from stage poses the major challenge for GPUs because it is not a top-down perspective and Section IV details each kernel well-suited to the SIMD paradigm. In this stage, the wavelet- employed. Section V evaluates the throughput of our archi- transformed image is partitioned in small sets of typically tecture and compares it to some of the fastest JPEG2000 and 64×64 wavelet coefficients, called codeblocks, and codes HEVC implementations. The last section contains conclu- them independently. This provides coarse-grain parallelism. sions. The coding within each codeblock must be carried out by a single thread, since there exist causal relationships among co- II. BACKGROUND efficients. This means that the coding of a coefficient depends A. NVIDIA GPU ARCHITECTURE on the output of the previous, so they can not be processed in Nvidia GPUs are hardware devices that are mainly consti- parallel. Even so, there have been efforts to implement this tuted by individual computing units called Streaming Mul- stage in GPUs [34]–[40], though these solutions do not to tiprocessors (SMs). Depending on the model and the ar- fully occupy the resources of the GPU due to the lack of fine- chitecture, a Nvidia GPU may contain from one to tens of grain parallelism. In 2014, we started a line of research [41]– SMs. Each SM can work independently, allowing the GPU to [45] focused on providing fine-grain parallelism to this stage process sequences of instructions from different algorithms. without sacrificing any feature of the system. The goal was Typically, SMs execute multiple 32-wide vector instructions not to implement the compliant JPEG2000 algorithm, but in parallel. to redevise it keeping in mind the SIMD architecture of CUDA refers vector instructions as warps. Each lane of 2 VOLUME VOL, YEAR Carlos de Cea et al.: GPU-oriented architecture for an end-to-end image/video codec based on JPEG2000 Lossy Compression Irreversible Irreversible Quantization CT DWT 9/7 RGB Compressed BPC + AC CR Image Data Reversible Reversible CT DWT 5/3 Lossless Compression FIGURE 1: JPEG2000 coding pipeline. a vector is virtualized into a software thread. Aggregations organize the execution of running kernels. Each stream may of 32 threads form a warp. A group of warps, called thread process a sequence of kernels of an application in a set of block, is assigned to a SM for execution. From the first SMs asynchronously from the rest. An appropriate use of the CUDA-compatible architecture (v1.0) up to Pascal (v6.2), streams optimizes the use of the GPU resources, which can warps are always executed synchronously and in a lock-step help to increase the throughput. fashion, featuring an implicit synchronization at the end of any divergence [50]. Volta (v7.0) introduced a modification B. JPEG2000 in the warp scheduler that allows the execution of warp As previously stated, JPEG2000 is an image/video coding threads asynchronously [51], so the synchronization among standard employed in professional environments due to its threads must be explicitly programmed when needed. Our excellent features and performance. The proposed codec codec is adapted to work with both implicit and explicit carries out almost the same operations as JPEG2000, so synchronization. they are briefly described herein for completeness. Figure 1 The memory architecture of the GPU is organized in depicts these operations. Depending on whether lossy or three levels: global, shared, and local. The size of the global lossless compression is needed, some of these operations are memory is, in general, in the order of GBs and is accessible irreversible or reversible. As stated before, the first stage of by all SMs. When this memory is accessed in a coalesced JPEG2000 applies several transformations to the image. The way (i.e., via consecutive positions) the available bandwidth first is carried out for color images, converting the red, green, is used efficiently and the latency is minimized. The size of and blue (RGB) components to the lesser redundant color the shared memory is in the order of MBs and its latency is space YCb Cr , which holds the luminance information in the lower than that of the global, though it can only be shared first component and the chrominance with respect to blue and within the thread blocks. The local memory is the fastest red in the second and third components, respectively. This though it is also limited in size and is only accessible by the is a pixel-wise operation that holds no dependencies among threads within a warp. The data allocated in the local memory pixels. It is carried out applying floating-point or integer are commonly stored in the registers, though they may be operations for the irreversible or reversible path, respectively. temporarily moved to the device memory (i.e., DRAM of The second operation is the DWT. Most implementations the GPU) when the register space is saturated. Typically, the apply it via the lifting scheme [55] since it has low com- global memory is employed to read and store the applica- putational complexity. The main idea behind this scheme is tion’s data, the shared memory is used for communication to first apply a series of arithmetic operations to all rows of among threads of different warps, and the local memory is the image and then to all columns. These operations can be utilized for intermediate computation. The local memory can carried out in parallel to all rows and then to all columns be shared among threads within a warp via the low-level since there are no inter-row/column dependencies. Then, the shuffle operation. This kind of memory sharing technique resulting coefficients are re-ordered taking the coefficients in proved to be very efficient in some applications [32], [52]– the even and odd positions in each direction. This produces [54]. The GPU has two levels of cache, denoted by L1 and four different subbands of one quarter the size of the original L2. The registers and the L1 cache are in the SM. The data image. In general, the same procedure is applied again four transferred from the device memory to the registers passes more times in the subband that contains the low-detail image. through the L1 and L2 caches, which are reservoirs of the The operations carried out in each step apply a low- and high- most recently accessed data to be (possibly) reused in future pass filter. JPEG2000 uses the irreversible CDF 9/7 and the petitions. reversible CDF 5/3. As previously mentioned, each SM runs thread blocks. The irreversible filter bank employs floating-point arith- These blocks can execute code from one or more CUDA metic, so the resulting coefficients need to be converted functions, called kernels, independently. This allows the par- to integers before bitplane coding. This operation is called allel execution of many different kernels from a single or deadzone quantization [8]. It multiplies the coefficients by various applications. CUDA provides the so-called streams to a step size and keeps the integer part. This operation is VOLUME VOL, YEAR 3 Carlos de Cea et al.: GPU-oriented architecture for an end-to-end image/video codec based on JPEG2000 not necessary for the reversible transform since it already replaced by the coding engine proposed in [44], [45]. The aim produces integer coefficients. of our codec is to code large quantities of images. The input The second main stage of the coding pipeline carries out data set may contain frames of a video sequence or images bitplane coding together with arithmetic coding. As stated of the same size. For convenience, frame is used to refer both before, this stage is applied in each codeblock independently. terms in the following. Through the binary representation of the integer coefficients When possible, the proposed architecture joins operations (without sign), a bitplane is defined as the set of bits from all in a single kernel instead of using a straightforward approach coefficients in the same binary position. Bitplanes are coded that uses one kernel per operation. Within the same kernel, from the most to the least significant. Just after the first non- the data are always accessed in the same fashion and the data zero bit of a coefficient is coded (referred to as significance types do not change. This permits the kernels to maximize the bit), its sign is coded too so that the decoder can reconstruct use of local memory in detriment of shared memory, using that coefficient. The bits coded for a coefficient after its a register-based strategy [52]–[54] that minimizes memory significance bit are called refinement bits. The coefficients latencies. When the data set needs to be re-organized or within a codeblock are scanned in a pre-defined order that the data type is changed, then the data are transferred to visits four rows of coefficients, called stripes, consecutively. the global memory preparing them for the next kernel. This In each stripe, coefficients are scanned from the left- to right- architecture minimizes the overall memory transfers and most column and, in each column, from the top to the bottom significantly increases performance. row. JPEG2000 codes each bitplane in three coding passes. Algorithm 1 describes the main routine of the codec. Its The first is called significance propagation. It follows the architecture is also illustrated in Figure 2. First, all memory scanning order processing only those coefficients that have needed during the coding process is pre-allocated both in the at least one significant neighbor. The second is called mag- host RAM and the device DRAM, which are respectively nitude refinement. It processes coefficients that were found referred to as MH and MD . This allocation (lines 1 and 2 in significant in previous bitplanes. The third pass processes the the algorithm) considers the space needed for a double buffer remaining coefficients. It is called cleanup. This multiple- strategy to load the frames (see below), auxiliary memory pass coding is aimed to code first the information that reduces structures, and number of GPU streams employed. The host the most the distortion of the image [6]. RAM allocation is performed in pinned memory2 to avoid Each processed bit is fed to the arithmetic coder together memory positions requests to the CPU when transferring with its contextual information. The context considers the data. This allocation greatly improves the memory bandwidth significance, or sign, of its eight neighbors. One of 18 achieved in some GPUs. See, for instance, in Table 1 the different pre-defined contexts is chosen depending on this difference in the bandwidth achieved by our codec when information. The context of the coefficient is employed by the coding a 4K video (with the test environment described in arithmetic coder to establish a probability for the currently Section V) using pinned or paged memory. To use pinned processed bit, generating a compacted stream of bits. memory in the Nvidia GTX 1080 Ti (Pascal architecture) The output produced in this stage for each codeblock is a almost doubles the bandwidth achieved as compared to paged bitstream that can be truncated at the end of each coding pass. memory. For the RTX 2080 Ti (Turing architecture), the Like most coding systems, JPEG2000 permits specifying a differences are much smaller due to the use of DDR4 RAM size for the final codestream, so bitstreams may be truncated modules in the host, though there is a slight increase of 4% in to fit the target rate. This rate-distortion optimization proce- the bandwidth achieved. It is worth noting that the practical dure is not defined in the standard, so each codec can choose maximum speed of the PCI-E 3.0 bus employed is 13.2 GB/s among a great variety of methods [56]. The final operation (with 15.8 GB/s of theoretical maximum), so our codec yields re-organizes these bitstreams to put them in the compressed maximum bandwidth in practice. file together with ancillary information for decoding. The Memory transfers are programmed to be asynchronous decoder carries out the same operations in reverse order so they can absorb variations in the time spent to process except the rate-distortion optimization stage, which is not each frame. The reading of frames is managed by a thread, necessary. denoted by t1 in Algorithm 1, that is executed by the host. Each stream, denoted by Sj , j ∈ {1..S} b with Sb being the III. OVERVIEW OF THE CODEC ARCHITECTURE number of streams, employs two input buffers in both MH A. OVERVIEW and MD so that when a buffer is being processed the other Except for bitplane and arithmetic coding, all operations of can be filled. These buffers are referred to as MH [i], MD [i] the JPEG2000 coding pipeline offer fine-grain parallelism. with i ∈ {1..2S}. b This filling is carried out in lines 3-6. t1 Our codec implements these operations following the stan- continuously checks if there is any empty buffer in MH . If dard, so their input/output is the same as that obtained so, it reads the data from disk and transfers them to MH . by a conventional JPEG2000 implementation. To use the Then, it issues an asynchronous copy to the device memory JPEG2000’s bitplane and arithmetic coder would signifi- cantly hinder the throughput of the GPU, so this is the only 2 Pinned memory indicates that the allocated space has a fixed location in stage that is not compliant with the standard. This stage is the RAM module(s) during the whole execution. 4 VOLUME VOL, YEAR Carlos de Cea et al.: GPU-oriented architecture for an end-to-end image/video codec based on JPEG2000 frame data transfer from host to device (and reverse) device frame data transfer within device auxiliary data transfer within device global memory D D M host M [i] A 1..3 Dk Bl [o] host R R' R R R 0 1 0 1 0 1 0 1 1 0 0 1 1 0 1 0 1 0 0 1 0 1 0 1 0 1 1 0 0 1 1 0 1 1 0 1 0 1 1 0 0 1 1 0 0 0 1 0 1 0 1 1 0 0 1 1 0 0 1 0 1 0 1 1 0 0 1 1 0 1 0 1 1 0 1 local mem. 1 1 0 1 1 1 0 1 1 1 0 1 0 1 0 0 1 0 0 0 0 1 1 1 0 1 1 1 0 1 0 1 0 1 1 1 0 1 1 1 0 1 0 1 1 0 0 1 1 0 1 1 1 0 1 0 1 0 1 1 1 0 1 1 1 0 1 0 1 0 1 1 0 0 1 0 0 1 0 0 0 1 0 0 0 1 0 0 0 1 0 1 1 1 1 0 0 0 0 1 0 0 0 1 0 0 0 0 1 0 0 1 0 0 0 1 0 0 0 0 1 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 1 1 0 1 1 0 0 1 1 0 0 1 0 0 1 0 1 0 1 1 0 1 0 1 0 1 1 0 0 1 0 0 1 0 1 1 1 1 1 0 0 1 0 0 1 0 1 1 1 0 1 1 0 0 1 0 0 1 0 1 1 1 1 1 0 0 1 0 0 1 0 1 1 1 1 0 0 0 stream 1 CT DWT_Q BPC_AC CR 1..3 H H H H M M [o+1] M M [i] [i+1] D [o] A M [o+1] D M [i+1] 1..3 Dk Bl es R R' R R R m 1 0 1 1 0 0 1 1 0 1 0 1 0 0 1 0 1 0 1 1 0 0 1 0 0 1 1 0 1 1 0 0 1 1 1 1 0 0 1 1 0 1 0 1 0 1 1 0 1 0 0 1 1 0 1 1 1 1 1 1 1 1 0 0 1 1 0 1 0 0 1 0 1 1 1 0 0 fra 0 1 0 1 1 0 1 0 1 0 0 1 0 0 0 0 1 1 1 1 0 0 1 1 0 1 0 1 0 1 1 1 0 0 0 1 1 0 1 0 1 0 0 0 1 0 0 1 1 1 0 1 0 1 0 1 0 0 1 1 1 1 1 0 1 0 1 0 1 1 0 1 1 0 1 1 0 0 1 1 0 0 1 0 0 0 1 0 1 1 1 1 0 0 1 0 1 0 1 0 0 1 0 0 0 1 1 1 0 1 1 1 0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 1 1 0 1 1 0 1 0 0 1 0 0 0 1 1 1 1 1 0 1 1 0 1 1 0 0 1 0 0 1 0 1 0 1 1 0 1 0 1 0 1 1 0 1 0 1 0 0 1 0 1 0 1 1 0 0 1 1 1 0 0 1 0 1 0 0 0 1 1 1 1 1 0 0 1 0 1 0 1 1 0 0 1 0 1 0 0 1 0 1 0 0 0 0 0 0 0 1 1 1 stream 2 original CT DWT_Q BPC_AC CR compressed data data main memory main memory disk disk R Y G Cb B Cr ~3.6% ~8.4% ~86% ~1.5% FIGURE 2: Illustration of the codec architecture when using 2 CUDA streams. The cycle of the data is as follows. First, frame data (individually identified by color) are read from disk to a RAM buffer. Then the data are managed by a stream in the GPU. Within the device the data are transferred from global memory MD to local memory R and inversely before and after running each kernel. The kernel execution is illustrated by the matrix of 0s and 1s. Each stream processes the three components of the frame before transferring the compressed data back to the host memory MH and disk. GTX 1080 Ti RTX 2080 Ti MH → MD MD → MH MH → MD MD → MH paged 7.5 GB/s 6.226 GB/s 12.746 GB/s 12.502 GB/s pinned 12.431 GB/s 12.725 GB/s 13.182 GB/s 13.175 GB/s speedup 1.65 2.04 1.03 1.05 TABLE 1: Evaluation of the memory bandwidth achieved by our codec when transferring data from host to device (MH → MD ) and device to host (MD → MH ) with pinned and paged memory, for two different GPUs. in line 5. t1 is active until all frames have been buffered. Algorithm 1 Main routine of the codec The data are read and stored considering their original bit- 1: CPUMemoryAllocation() depth to optimize transfers and memory space. In general, 2: GPUGlobalMemoryAllocation() 8-bit integers are employed. 3: for each empty MH [i] do The writing of the compressed data to the disk is done 4: MH [i] ← HDRead() similarly by thread t2 , which is executed by the host in t1 5: MD [i] ← MH [i] lines 13-16. A double-buffer strategy is also employed so 6: end for that when a stream finishes coding a frame, it can readily 7: A1..3 ← CT(MD [i]) start coding another without waiting for the compressed data 8: for k ∈ {1..3} do to be transferred to the host memory. These output buffers 9: Dk ← DWT_Q(Ak ) are referred to as MH [o], MD [o] with o ∈ {1..2S}. b Again, Sj 10: {Bl } ← BPC_AC(Dk ) the data transfer from device to host is carried out via an 11: MD [o] ← CR({Bl }) asynchronous copy in line 14. Once the transfer is done, 12: end for t2 writes them to the disk. The data are copied in the disk 13: for each filled MD [o] do orderly, i.e., following the same frame order of the original 14: MH [o] ← MD [o] sequence. t2 15: HDWrite(MH [o]) Lines 7-12 in Algorithm 1 describe the calls to the ker- 16: end for nels and the auxiliary memory structures employed in the GPU. Four kernels are used. The first carries out the color transform. It transfers all frame data from MD [i] to local memory converting them to 32-bits integers (floats) for the kernel. After this, each component is processed indepen- (ir)reversible path and performs the arithmetic operations dently. The next kernel carries out the DWT and, if using on the registers. The result is left in the auxiliary structure lossy compression, quantization. Our codec employs a rate- denoted by A1..3 using the same data type employed in the distortion optimization method that controls the rate through VOLUME VOL, YEAR 5 Carlos de Cea et al.: GPU-oriented architecture for an end-to-end image/video codec based on JPEG2000 occupancy warp efficiency bandwidth (GB/s) time (µs) #inst. (×106 ) #inst. per sample 2K 4K 2K 4K 2K 4K 2K 4K 2K 4K 2K 4K CT(·) 90% 87% 100% 100% 483 495 65 255 3.08 12.32 1.47 1.47 DWT_Q(·) 84% 90% 97.5% 97.5% 471 511 36 135 2.45 9.81 1.17 1.17 BPC_AC(·) 18% 61% 63% 63% 69 189 1150 2000 32.61 107.01 15.54 12.75 CR(·) 88% 88% 99% 99% 181 210 15 35 0.82 3.05 0.39 0.36 TABLE 2: Analysis of the codec’s kernels when coding a 2K and 4K frame with the Nvidia RTX 2080 Ti. registers data reading (MB) data writing (MB) per thread MD → R L2 → L1 L1 → L2 R → MD 2K 4K 2K 4K 2K 4K 2K 4K 2K 4K CT(·) 18 18 6 24 6 24 24 96 24 96 DWT_Q(·) 63 63 8.22 33.89 14.93 61.05 9.95 40.18 8.21 32.39 BPC_AC(·) 60 60 48.59 251.79 67.01 293.38 37.26 129.58 26.95 108.97 CR(·) 24 24 1.04 3.24 1.11 3.46 0.64 2.01 0.87 2.75 TABLE 3: Analysis of the hierarchical memory transfers of the codec’s kernels when coding a 2K and 4K frame with the Nvidia RTX 2080 Ti. the quantization step employed in this operation [56]. It samples. We recall that this kernel processes the three image transfers the data from Ak to the registers, applies the lifting components, whereas the following kernels process only scheme, and leaves the result in Dk . The third kernel is the one. As seen in Table 3, the three image components are most complex. It applies bitplane and arithmetic coding. Like transferred from MD to R requiring 6 and 24 MB for a 2K the other kernels, it reads the data from the global memory and 4K frame, respectively. Once the data are in the SM, they and puts them in the local. These data are organized in are converted from 8-bit integers to 32-bit integers or floats codeblocks holding 64×64 coefficients. Each codeblock is depending on whether the reversible or irreversible transform processed by an individual warp of 32 threads. The result of is selected. This conversion is seen in the memory transfers this kernel is stored in the set {Bl } that contains one bitstream when the data are transferred back from the registers to the per codeblock, with l ∈ {1..L} b and L b being the number of device memory via the L1 and L2 caches. codeblocks per component. The length of each bitstream is The DWT_Q(·) kernel can perform a variable number of not known before coding, so the space for bitstreams {Bl } transformation levels, typically 5. The metrics reported in Ta- is pre-allocated amply. As a result, the bitstream data are ble 2 correspond to the first call to the kernel, which performs scattered throughout the whole structure. These data must be the first level of transformation. The achieved occupancy compacted before transferring them to the host memory and is about 84% for 2K and 90% for 4K. This indicates that disk, which is the function of the last kernel. Contrarily to the other computations can be done while this kernel is running. other kernels, it does not put the frame data to the registers but Similar to the previous kernel, the warp efficiency is almost only the lengths of the generated bitstreams (via pointers to 100% since there are no divergent paths. The increase in memory positions), so that it can compute the final position of execution time and total number of instructions between 2K each compressed byte. Then, it re-organizes the compressed and 4K is also proportional to the frame size. As seen in frame data in the global memory leaving them in one of the Table 3, this kernel utilizes more registers per thread due to two output buffers. a larger data tile processed by each warp. The data require 8 The decoder employs a similar structure to that of the MB and 32 MB for the 2K and 4K frame, respectively, which encoder. It executes the kernels in inverse order, performing approximately correspond to the transfers between MD to the reverse operations. R and inversely. The extra data transferred correspond to auxiliary information. The transfers between the L1 and B. ANALYSIS L2 cache are higher than those from the device memory Table 2 and 3 report the kernels’ metrics obtained via the to the registers because this kernel processes the data tiles Nvidia Nsight Compute tool when coding a 2K and 4K frame employing a redundant halo. using the test environment described in Section V. The first As shown by the metrics, the BPC_AC(·) kernel is the kernel (i.e, ICT(·)) achieves high occupancy, optimal warp most complex. First, the occupancy is much lower than that efficiency (since it does not have divergence), and very high achieved by the other kernels, especially for 2K frames. This memory bandwidth (see Table 2). These results are due to is because 2K frames do not have enough data to fill the the pixel-wise operation that it carries out. The differences resources of the GPU. 4K frames achieve higher occupancy, between the 2K and 4K frame with respect to execution though it is still below that achieved by the other kernels. time and total number of instructions executed are a 4 fold Second, the warp efficiency is 63% due to the multiple diver- increase, coinciding with the increase in number of processed gent paths of the algorithm. Third, the memory bandwidth 6 VOLUME VOL, YEAR Carlos de Cea et al.: GPU-oriented architecture for an end-to-end image/video codec based on JPEG2000 is much lower than that achieved by the other kernels since 8000 BPC_AC(·) is bounded by the latency of the computing 7000 instructions [45]. Fourth, the time spent for coding a 2K 6000 and 4K frame is not proportional to the frame size. This is due to the low occupancy that is achieved for 2K frames 12K 5000 and due to the image content. Let us explain further. The MS/s 4000 codeblock size is 64×64 regardless of the frame size. This causes that codeblocks of 2K frames have more details (i.e., 3000 more entropy) than codeblocks of 4K frames, requiring more 2000 8K instructions to code their information. This is manifested in Encoder 4K Encoder 2K the total number of instructions and instructions per sample 1000 4K executed, since the 4K frame requires approximately 20% 0 fewer instructions to code each sample. The memory trans- 5 10 15 20 fers when reading the data are higher than in the other kernels #streams mainly due to the register pool size (see Table 3). Differently FIGURE 3: Analysis of the throughput achieved by the pro- from the previous kernels, BPC_AC(·) visits each coefficient posed codec when encoding 2K and 4K video using different of the codeblock many times. The number of visits depends number of execution streams, for the RTX 2080 Ti. on the codeblock’s data, but is approximately 8 or 9 times per coefficient on average. Since the size of the register space Algorithm 2 Kernel routine CT(MD [i]) is limited, once a coefficient is visited it is transferred back to the device memory so the register can be employed for 1: GPULocalMemoryAllocation() other coefficients. When the coefficient is needed again, it is 2: R1..3 ← MD [i] transferred from the device memory to the registers. Many 3: R01..3 ← φ(R1..3 ) of these coefficients are kept in cache and are reused, so 4: A1..3 ← R01..3 the transfers between the L2 and L1 cache are high as well. 5: return(A1..3 ) The data transfers when writing are not as high because the kernel only stores the compressed data. Even so, the data in the compressed bitstream are accessed many times, so the straight horizontal lines for the convenience of the reader. transfers between registers and device memory are higher As seen in the figure, the throughput increases notably when than in the previous kernels. multiple streams are employed. In the case of 2K (4K) video, The occupancy and efficiency of the CR(·) kernel is similar 13∼14 (7∼8) streams obtain maximum efficiency. Again, the to that achieved by ICT(·) and DWT_Q(·). The execution coding of 4K video achieves higher throughput due to the time for 4K frames is twice as that needed for 2K. This nature of the data. is because both frames require 5µs to generate preliminary As seen in Section V the throughput achieved by the de- tables, and then the data to be reorganized are about 1 MB coder is only slightly lower than that of the encoder because and 3 MB respectively for the 2K and 4K frame,3 requiring the decoder requires more local memory, which reduces the 10µs and 30µs. The memory bandwidth is lower than that occupancy. The rest of the decoding process is very similar obtained in the first two kernels since the transferred data are to that of the encoder, so it is not reported herein for brevity. already compressed(also seen in Table 3). This analysis indicates that the BPC_AC(·) kernel con- IV. DESCRIPTION OF THE KERNELS sumes most of the total execution time and it achieves the Algorithm 2 details the routine of the CT(·) kernel. In this lowest occupancy. This suggests that the codec may underuse and following kernels, the algorithm describes the main op- the resources of the GPU when coding large sets of images erations that are performed at a thread level. Like in the other or video unless more workload is feed to the device. The pro- kernels, the first instruction allocates the local memory. All posed architecture alleviates this issue by employing multiple kernels only use registers since this increases the throughput. streams of execution. Each stream processes a frame, so more After allocating the required space, the data of the three data are processed in parallel, employing more resources frame components are transferred from the global memory and increasing the overall throughput. See in Figure 3 the to the register space, referred to as R for the input data. throughput achieved by our multiple-streamed codec when This is the only kernel that needs the three components of encoding 2K and 4K video in the same conditions as before. the frame. It applies a transformation that involves several The results are reported as the number of Mega Samples arithmetic operations, denoted by φ(·) in line 3, and the result coded per second (MS/s). The figure depicts the throughput is left in the output register space R0 . Then the data are needed to code 4K, 8K, and 12K video in real-time with returned to the global memory, ready to be fetched by the next kernel. Both reading and writing in the global memory 3 4K frames are compressed more efficiently than 2K frames, so they in this and following kernels is carried out in a coalesced generate fewer data per sample coded. way to maximize memory performance since the GPU stores VOLUME VOL, YEAR 7 Carlos de Cea et al.: GPU-oriented architecture for an end-to-end image/video codec based on JPEG2000 Algorithm 3 Kernel routine DWT_Q(Ak ) the global memory. It is only done for those samples that 1: GPULocalMemoryAllocation() do not belong to the halo.5 Before transferring the data, a 2: R ← Ak quantization step size, denoted by Q in line 16, may be 3: for y ∈ {1..Yb } do applied. Again, lossy and lossless compression respectively 4: for x ∈ {0..1} do requires the use of floating points and integers when applying 5: R[y][x] ← ϕ(R[y][x]) ϕ(·). Quantization is only applied for lossy compression. 6: end for The BPC_AC(·) kernel is detailed in Algorithm 4. It is 7: end for applied to all codeblocks of the component, though we recall 8: for x ∈ {0..1} do that the algorithm details the operations carried out at thread 9: for y ∈ {1..Yb } do level. The kernel receives a frame component that is parti- 10: R[y][x] ← ϕ(R[y][x]) tioned in codeblocks of 64×Yb 0 coefficients, with typically 11: end for Yb 0 = 64. The data for the codeblock are implicitly trans- 12: end for ferred to the local memory in line 2 of the algorithm. Then the 13: for y ∈ {1..Yb } do coefficients are coded from bitplane B, b which is a sufficient 14: for x ∈ {0..1} do number of magnitude bits to code all coefficients within the 15: if (y, x) ∈ / halo then codeblock, to the lowest bitplane 0. This is performed in 16: R[y][x] ← R[y][x] · Q the loop of line 3. Like in the previous kernel, each thread 17: Dk ← R[y][x] processes two columns and each codeblock is processed by a 18: end if warp. Contrarily to JPEG2000, this kernel carries out 2 cod- 19: end for ing passes instead of 3 since virtually same compression effi- 20: end for ciency is achieved [6], [44], [45] while increasing throughput 21: return(Dk ) about 40%. The loop in lines 4-17 performs significance coding. It checks whether the coefficient was significant in previous bitplanes via the γ(·) function, which returns the significance bitplane of the coefficient. If not, significance data blocks adjacent to that requested in the L2 cache for coding is performed. First, context C of the coefficient is (possible) future requests. Depending on whether lossy or determined via Φ(·) and, through this context and the current lossless compression is selected, the operations and the data bitplane, probability P for the coded bit is extracted from types employed in the registers are floating points or integers, the lookup table LUTsig . This table contains pre-computed respectively. probabilities determined with a training set of images. Then, The second kernel is detailed in Algorithm 3. The wavelet the bit is coded via arithmetic coding. The procedure for transform is applied in blocks of 64×Yb samples that are AC(·) is not detailed in the algorithm for simplicity. It can processed by a single warp.4 This allows communication be found in [45]. If the coefficient is significant in the current among threads without needing shared memory. The height bitplane (i.e., γ(R[y][x]) = b), its sign is coded in lines 11-13 of the block is denoted by Yb . Each thread processes two with a similar procedure to that of significance coding. Re- columns of a block. The kernel applies a 2D high-pass/low- finement coding is carried out in lines 18-25. In this case, no pass filter to all samples. First, the filter ϕ(·) is applied context is employed. The return of the AC(·) function is the horizontally (lines 3-7) and then vertically (lines 8-12). The bitstream Bl that contains the compressed information. Each filter consists in a series of arithmetic operations that use the time that this function is called, some data may be added to adjacent samples to the processed coefficient, in which the Bl . We note that Bl is in the global memory. Each thread result is left. This type of operation does not require two puts data in Bl asynchronously from the others ensuring register spaces (for input and output) like in the previous mutual exclusion. This exclusion is guaranteed considering kernel, but only one that is referred to as R. When the thread the threads that need a new chunk of memory to write their needs data from other threads, it uses shuffle instructions information, assigning positions based on the thread index (not shown in Algorithm 3) since they have lower latency within the warp. This kernel also stores the length of Bl in a than using shared memory [32]. If more than one level of separate global memory region, denoted by L. wavelet transform is selected, the instructions from line 3 The last kernel (i.e., CR(·)) is detailed in Algorithm 5. It to 12 are repeated each time over a quarter of the last data receives the set of bitstreams {Bl }. As previously stated, its processed, which contains the results of the low-pass filter. purpose is to reorganize the bitstream data in a compact struc- This is carried out calling the kernel again. It is not detailed ture. To do so, blocks of 2 bytes are assigned to each thread in in Algorithm 3 for the sake of clarity. The final step in the warp to be written in the final memory positions. The first this routine is to transfer the data from the local space to step is to generate a memory map to know these positions. 4 Note that these blocks are not the codeblocks utilized in BPC_AC(·), but This map is denoted as L0 and contains an aggregated list of a tile of the original image. Although the partitioning is similar for paral- lelism purposes, the block transformed by DWT_Q(·) contains overlapped 5 The halo is an area surrounding the processed samples that is employed samples of adjacent blocks. by the warp to obtain the correct result of the wavelet transform. 8 VOLUME VOL, YEAR Carlos de Cea et al.: GPU-oriented architecture for an end-to-end image/video codec based on JPEG2000 Algorithm 4 Kernel routine BPC_AC(Dk ) or compressed data. This is carried out in line 2 checking 1: GPULocalMemoryAllocation() if the thread is copying the first bytes of the codeblock’s 2: R ← Dk bitstream. The corresponding bytes are either copied to the 3: for b ∈ {B..0} b do header or body section of the final structure, respectively 4: for y ∈ {1..Yb 0 } do denoted by MD [o][H] and MD [o][D]. The data transfers are 5: for x ∈ {0..1} do also performed in a coalesced fashion to maximize through- 6: if γ(R[y][x]) ≤ b then put. 7: C ← Φ(R[y][x]) Again, the kernels employed in the decoder are very simi- 8: P ← LUTsig [C][b] lar to those of the encoder, so they are not detailed herein. 9: Bl ← AC(R[y][x], P ) 10: if γ(R[y][x]) = b then V. EXPERIMENTAL RESULTS 11: C 0 ← Φ0 (R[y][x]) The proposed codec is evaluated with four Nvidia GPUs, 12: P 0 ← LUTsign [C 0 ][b] namely, the RTX 2080 Ti, the GTX 1080 Ti, the Xavier, 13: Bl ← AC(R[y][x], P 0 ) and the Tegra X2. These devices are commodity GPUs, with 14: end if prices ranging from 650 C to 1350C. Their specifications are 15: end if reported in Table 4. Both the RTX 2080 Ti and the GTX 16: end for 1080 Ti are commonly employed in workstations for design 17: end for applications and gaming. The RTX 2080 Ti has the highest 18: for y ∈ {1..Yb 0 } do peak throughput. It is employed with an i9 9900K CPU 19: for x ∈ {0..1} do workstation with 16 GB of DDR4 RAM. The 1080 Ti is used 20: if γ(R[y][x]) > b then on an i7-3770 workstation with 8 GB of DDR3 RAM. Both 21: P 00 ← LUTref [b] the Xavier and the Tegra X2 are GPUs devised for devices in 22: Bl ← AC(R [y][x], P 00 ) which efficiency and size are important aspects, for example 23: end if in the Nintendo Switch. In our tests, they run on a Jetson 24: end for SDK platform [58]. Both GPUs have low performance, but 25: end for consume very little power. Both allow different power modes 26: end for with varying performance and Thermal Design Power (TDP). 27: Ll ← length(Bl ) The results reported below correspond to the maximum per- 28: return(Bl ) formance mode except when indicated. JPEG2000 results are obtained with Kakadu (v8.0.2) [59]. Kakadu is among the fastest CPU implementations of the Algorithm 5 Kernel routine CR({Bl }) standard. It is heavily optimized in assembler, achieving su- 1: S ← computePosition(T, LUTL0 , L0 ) perior throughput than other implementations for GPUs such 2: if (S ∈ {L0 }) then as CuJ2K [60] and GPU-J2K [61]. It is executed in a work- 3: MD [o][H] ← Bl [S] station with an Intel i9-9900K CPU with 8 cores and 16 GB 4: else of DDR4 RAM. Kakadu is compiled for this architecture and 5: MD [o][D] ← Bl [S] it is run with 16 threads of execution to achieve maximum 6: end if throughput. The compression parameters for both Kakadu 7: return(MD [o]) and our codec are: lossy or lossless compression as indicated, 5 levels of DWT, and codeblocks of 64×64. Although there are other competitive GPU implementations of JPEG2000 such as Comprimato [62] and CUDA-JPEG2000 [63], it was lengths, more precisely, L0 = {0, L1 , L1 + L2 , · · · , L1 + not possible to compare them in our test environment. Some · · · + LLb }. L0 is generated via the Device Scan primitive results reported in their corresponding webpages suggest that from the Nvidia CUB framework [57]. To accelerate the they obtain competitive throughput, though lower to that access to this map, a fast lookup table, denoted by LUTL0 , is achieved by the proposed codec. created. This LUT is generated applying a binary search over For comparison purposes, the following experiments also L0 in which each position represents some positions of the provide the throughput achieved with the HEVC implemen- original map. Our experience indicates that speedups about tation developed by Nvidia [64], which is executed with the 2× are achieved by using such a strategy. These operations RTX 2080 Ti and the GTX 1080 Ti. This codec runs in are carried out before running the CR(·) kernel, so they are the GPU employing in-chip support and dedicated hardwired not specified in Algorithm 5. components. The parameters for HEVC are: rate control Once the LUTL0 is created, each warp thread T computes with constant quantization 1-51 (0) for lossy (lossless), inter- the position S of the data to be written (line 1). Then, it frame coding with GOP=32, and high performance mode. checks whether the information to be copied is auxiliary This configuration achieves maximum throughput in our information of the codeblock (i.e., most significant bitplane), tests. We note that HEVC is not supported in Jetson GPUs. VOLUME VOL, YEAR 9 Carlos de Cea et al.: GPU-oriented architecture for an end-to-end image/video codec based on JPEG2000 cores clock memory peak FP32 compute memory SMs × SM frequency bandwidth throughput capability TDP size RTX 2080 Ti 68 64 1601 MHz 616 GB/s 13.935 TFlops 7.5 (Turing) 260 W 11 GB GTX 1080 Ti 28 128 1923 MHz 484 GB/s 13.78 TFlops 6.1 (Pascal) 250 W 11 GB Xavier 8 64 854∼1377 MHz 137 GB/s 1.4 TFlops 7.2 (Volta) 10/15/30 W 16 GB? Tegra X2 2 128 854∼1465 MHz 58.4 GB/s 0.75 TFlops 6.2 (Pascal) 7.5 - 15 W 8 GB? TABLE 4: Features of the GPUs employed. ? Both the Xavier and Tegra X2 do not have dedicated GPU memory. Memory is shared by both the CPU and GPU. 10000 9000 Encoder 2080 Ti Decoder 2080 Ti Encoder 1080 Ti Decoder 1080 Ti 9000 Encoder Xavier Decoder Xavier 8000 2K 4K Encoder X2 Decoder X2 8000 7000 7000 6000 6000 12K MS/s 5000 5000 4000 4000 3000 3000 2000 8K 2000 1000 1000 4K 0 0 5 10 15 20 5 10 15 20 #streams FIGURE 4: Analysis of the throughput achieved by the proposed codec when coding 2K (left) and 4K (right) video using different number of execution streams, for lossy compression at maximum quality. The data set employed in the experiments is a 2-minute size that achieves maximum quality (about 50 dB). Figure 4 segment of the movie “Star Wars: The Last Jedi,” at a reports the results achieved. Again, this figure depicts with resolution of 2K and 4K. The video contains 2,880 color horizontal lines the throughput needed to yield 4K, 8K, and frames with a bit-depth resolution of 24 bits per pixel (i.e., 12K video compression in real time, assuming a frame rate 8 bits per pixel per component), resulting in 67,5 GB (16,875 of 24 frames per second. The results indicate that both the GB) of uncompressed data for the 4K (2K) resolution. The RTX 2080 Ti and GTX 1080 Ti increase the throughput as HEVC codec uses a subsampled 4:2:0 version of the video more streams are employed, yielding optimal performance for compatibility issues with the 4K resolution in the GTX depending on the frame resolution and GPU employed. The 1080 Ti. This is taken in consideration when measuring the Xavier and Tegra X2 do not benefit as much of using multiple performance achieved. In general, the size of this data set is streams because they have fewer SMs, so their resources are sufficiently large to fill the resources of the GPU. Larger data mostly filled with a single execution stream. In all results, sets achieve similar results as those reported below. In all the decoder yields slightly lower throughput than the encoder results, the execution time is measured without considering because it requires more local memory. This behavior is not the I/O time spent to read/write the files from/to the disk common in software implementations of image and video since that would affect results significantly depending on codecs since the encoder generally requires more compu- the hard drive employed. The results below evaluate only tations. Highly optimized implementations such as the pre- the throughput achieved since coding performance of the sented herein, however, may obtain different results due to proposed codec is extensively analyzed in [44]. Herein, the the need of different data structures in the decoder. In the codecs are compared when their coding options yield equiv- following tests, 20 and 9 streams are employed for the RTX alent image quality. 2080 Ti and GTX 1080 Ti, respectively, to achieve maximum throughput. The Xavier and Tegra X2 employ 14 and 10 The first test evaluates the throughput achieved by the streams, respectively, though their throughput is almost the proposed codec with the four GPUs when using a different same as when using only 2. number of execution streams. The test evaluates both the encoder and decoder in lossy mode with a quantization step The next test evaluates the number of kernels that are 10 VOLUME VOL, YEAR Carlos de Cea et al.: GPU-oriented architecture for an end-to-end image/video codec based on JPEG2000 12 mode. This throughput is similar to that obtained by Kakadu, though we recall that Kakadu employs a modern CPU and the 10 Xavier is an embedded mobile solution. Both for the Xavier 2K and the Tegra X2, the minimum power mode significantly 8 #kernels running lowers performance and the mid-tier mode achieves an inter- 6 mediate performance. This is more pronounced in the Xavier. HEVC yields higher performance than Kakadu, though it 4 4K is lower than that achieved by our codec. Surprisingly, the HEVC encoder achieves higher throughput with the GTX 2 Encoder 2080 Ti 1080 Ti than with the RTX 2080 Ti. Even though it is Decoder 2080 Ti executed using the Nvidia SDK HEVC software (v9.0) [64] 0 5 10 15 20 in maximum performance mode in both, each GPU has its #streams own hardwired solution for this codec. More precisely, the RTX 2080 Ti includes one NVEnc Turing engine whereas the FIGURE 5: Evaluation of the average number kernels exe- 1080 Ti includes two Pascal engines. Note also that the GTX cuted per unit of time depending on the number of streams 1080 Ti obtains higher throughput for the encoder than for employed. the decoder, whereas the RTX 2080 Ti yields more balanced results. The previous test evaluates the performance achieved executed in parallel depending on the number of streams when there is (almost) no quality loss. Scenarios such as employed. This analysis complements the previous for the video streaming or TV broadcast may tolerate more distor- RTX 2080 Ti. The GTX 1080 Ti, Xavier, and Tegra X2 are tion. Reducing the image quality results in higher throughput not included in this analysis. Figure 5 depicts the results since fewer data are coded. Figure 7 depicts the throughput achieved. For 4K video, the maximum number of running achieved by Kakadu, HEVC, and the proposed codec when kernels is 4, which is yield when employing 10 streams. coding 4K video at different levels of quality, namely, from 4 parallel kernels already fill the resources of the GPU. 50 dB to 20 dB, which is the quality range employed in This indicates that no more kernels can be executed despite most scenarios. The image quality is controlled via the quan- increasing the number of streams employed, although a slight tization parameter Q in our codec, and similarly in HEVC increase in throughput can be achieved as it seen in the pre- and Kakadu. As seen in the figure, reducing the quality has vious figure. 2K video obtains a different behavior. Number a direct impact on throughput for all codecs. The proposed of streams and running kernels are almost directly related, codec achieves real-time encoding of 16K video for qualities reaching a peak at 20 streams and 10 parallel kernels. This below 46 dB. The decoder has a lower increase in perfor- is because 2K frames have only a quarter of the data of 4K mance as the quality decreases because the aforementioned frames, so the GPU requires more kernels to fill its resources. need of more local memory. The Xavier and Tegra X2 also Figure 6 reports the throughput achieved by the proposed increase their throughput, though more gradually due to their codec with the four GPUs, Kakadu, and HEVC when coding inferior performance power. It is worth noting that, even 4K video in lossy and lossless mode. For lossy compression, though the RTX 2080 Ti and GTX 1080 Ti have a similar the average image quality yield for all codecs is about 50 peak throughput (about 14 TFlops), the RTX 2080 Ti obtains dB. At this level of quality, distortion is not perceptible by approximately 50% more throughput when encoding. This is the human eye. Each codec has a pair of columns. The first due to the distribution of performance power in the GPU. The reports the results for the encoder whereas the second for RTX 2080 Ti has fewer CUDA cores in each SM, but more the decoder. The results for 2K video are similar but with than twice SMs than the GTX 1080 Ti. This provides more lower performance, so they are not included in this figure. resources per thread, especially, more local memory. Our Results for the Xavier and Tegra X2 are reported when codec greatly benefits from this architectural improvement using three power modes, namely, maximum (0), minimum since it employs registers extensively. The highest speedups (1), and mid-tier (2) performance. The results show that the reported in Figure 6 are achieved by the HEVC decoder, proposed codec yields superior performance to that achieved which increases the throughput almost 6×. by Kakadu and HEVC for both the RTX 2080 Ti and GTX Power consumption is nowadays an important aspect due 1080 Ti regardless of using lossy or lossless compression. In to the advent of mobile devices. Figure 8 evaluates the power all codecs, the performance in lossless mode is slightly lower consumption of our codec, HEVC, and Kakadu when coding than that achieved in lossy since more data are processed, 4K video at 50 dB, like in Figure 6. The results are depicted generating larger compressed files. Even so, real-time 12K in MS processed per Watts consumed. A Nvidia tool that video can be managed by our codec for both compression measures consumption in real-time is employed to obtain modes. The Xavier and Tegra X2 GPUs do not achieve such a these results. Kakadu’s consumption is measured via the high performance, but the Xavier is able to process 4K video utility PowerTOP. The results depicted in Figure 8 suggest in real time when employing the maximum performance that the proposed codec is the most efficient in terms of power VOLUME VOL, YEAR 11 Carlos de Cea et al.: GPU-oriented architecture for an end-to-end image/video codec based on JPEG2000 9000 9000 8000 lossy lossless 8000 7000 7000 6000 6000 12K 5000 5000 MS/s 4000 4000 3000 3000 8K 2000 2000 1000 4K 1000 0 0 Kakadu 2080Ti 1080Ti 2080Ti 1080Ti Xavier(0,1,2) X2(0,1,2) Kakadu 2080Ti 1080Ti 2080Ti 1080Ti Xavier(0,1,2) X2(0,1,2) JPEG2K HEVC proposed JPEG2K HEVC proposed FIGURE 6: Throughput evaluation for lossy (with highest image quality) and lossless compression of 4K video, for all codecs and GPUs. Each pair of columns reports the results for the encoder (back) and decoder (front). 14000 14000 2080 Ti 2080 Ti HEVC 12000 Encoder Decoder 1080 Ti 1080 Ti HEVC 12000 Xavier Kakadu X2 10000 16K 10000 8000 8000 MS/s 6000 12K 6000 4000 4000 8K 2000 2000 4K 0 0 50 45 40 35 30 25 20 50 45 40 35 30 25 20 PSNR (in dB) FIGURE 7: Throughput evaluation for lossy compression of 4K video at different quality levels. Results are for the proposed codec except when indicated. consumption. Evidently, the Xavier and Tegra X2 yield the VI. CONCLUSIONS best results due to its architecture. Our codec employed with Faster and less power-hungry image and video codecs the three power modes of these GPUs is less power-hungry are currently needed in multiple scenarios. Typically, high than the remaining, with the minimum mode achieving the throughput codecs are achieved by means of integrated hard- highest efficiency. The proposed codec is more efficient than ware architectures such as ASICs or FPGAs. GPUs are also HEVC even when executed in the RTX 2080 Ti and GTX a widely pursued means to accelerate codecs, though these 1080 Ti, though moderately so. In general, CPUs consume architectures do not commonly obtain the high performance more power than GPUs, so Kakadu seems to consume the of their counterparts. This is because the core algorithms most. The low power consumption of our codec means that, of conventional image and video coding systems do not in practice, it can allow batteries of mobile devices last much provide enough fine-grain parallelism to fully exploit the longer and/or code more minutes of video for the same SIMD architecture of GPUs. This paper introduces an im- battery capacity. age/video codec based on the JPEG2000 standard. All stages of the coding pipeline have been devised to extract fine-grain parallelism. All stages are compliant with the standard except for the core algorithm called bitplane and arithmetic coding. 12 VOLUME VOL, YEAR Carlos de Cea et al.: GPU-oriented architecture for an end-to-end image/video codec based on JPEG2000 700000 implementation of JPEG2000 codec,” IEEE Trans. Consum. Electron., vol. 50, no. 2, pp. 638–646, May 2004. [13] H.-C. Fang, Y.-W. Chang, T.-C. Wang, C.-J. Lian, and L.-G. Chen, “Par- 600000 allel embedded block coding architecture for JPEG 2000,” IEEE Trans. Circuits Syst. Video Technol., vol. 15, no. 9, pp. 1086–1097, Sep. 2005. 500000 [14] G. Pastuszak, “A high-performance architecture for embedded block cod- ing in JPEG 2000,” IEEE Trans. Circuits Syst. Video Technol., vol. 15, MS / Watts 400000 no. 9, pp. 1182–1191, Sep. 2005. [15] A. K. Gupta, S. Nooshabadi, D. Taubman, and M. Dyer, “Realizing low- cost high-throughput general-purpose block encoder for JPEG2000,” IEEE 300000 Trans. Circuits Syst. Video Technol., vol. 16, no. 7, pp. 843–858, Jul. 2006. [16] Y. Li and M. Bayoumi, “A three-level parallel high-speed low-power 200000 architecture for EBCOT of JPEG 2000,” IEEE Trans. Circuits Syst. Video Technol., vol. 16, no. 9, pp. 1153–1163, Sep. 2006. 100000 [17] Y.-W. Chang, C.-C. Cheng, C.-C. Chen, H.-C. Fang, and L.-G. Chen, “124 MSamples/s pixel-pipelined Motion-JPEG 2000 codec without tile memory,” IEEE Trans. Circuits Syst. Video Technol., vol. 17, no. 4, pp. 0 Kakadu 2080Ti 1080Ti 2080Ti 1080Ti Xavier(0,1,2) X2(0,1,2) 398–406, Apr. 2007. JPEG2K HEVC proposed [18] K. Mei, N. Zheng, C. Huang, Y. Liu, and Q. Zeng, “VLSI design of a high-speed and area-efficient JPEG2000 encoder,” IEEE Trans. Circuits FIGURE 8: Power consumption evaluation when encoding Syst. Video Technol., vol. 17, no. 8, pp. 1065–1078, Aug. 2007. 4K video at 50 dB. Each pair of columns reports the results [19] Y.-W. Chang, H.-C. Fang, C.-C. Chen, C.-J. Lian, and L.-G. Chen, “Word- for the encoder (back) and decoder (front). level parallel architecture of JPEG 2000 embedded block coding decoder,” IEEE Trans. Multimedia, vol. 9, no. 6, pp. 1103–1112, Oct. 2007. [20] M. Dyer, S. Nooshabadi, and D. Taubman, “Design and analysis of system on a chip encoder for JPEG2000,” IEEE Trans. Circuits Syst. Video The proposed codec introduces a similar algorithm to that of Technol., vol. 19, no. 2, pp. 215–225, Feb. 2009. [21] K. Sarawadekar and S. Banerjee, “An efficient pass-parallel architecture JPEG2000 that augments its parallel capabilities. Although for embedded block coder in JPEG 2000,” IEEE Trans. Circuits Syst. the resulting codestream is not compliant with JPEG2000, Video Technol., vol. 21, no. 6, pp. 825–836, Jun. 2011. the coding system has the same advanced features of the [22] K. Liu, E. Belyaev, and Y. Li, “A high throughput JPEG2000 entropy decoding unit architecture,” SPRINGER Journal of Digital Imaging, 2019, standard. The throughput of the resulting architecture when in Press. executed in consumer-grade GPUs is at least 10× higher than [23] Nvidia. (2018, Dec.) GPU vs CPU theoretical GFLOP/s. [Online]. Avail- that achieved with CPU implementations executed in high- able: https://docs.nvidia.com/cuda/cuda-c-programming-guide/graphics/ end workstations, and superior to that achieved by Nvidia’s floating-point-operations-per-second.png [24] M. Hopf and T. Ertl, “Hardware accelerated wavelet transformations,” in SDK implementation of the HEVC video standard. Exper- Proc. Eurographics and IEEE Symposium on Visualization, May 2000, pp. imental results suggest that our codec can encode (decode) 93–103. real-time 12K (8K) video in a Nvidia RTX 2080 Ti and that [25] T.-T. Wong, C.-S. Leung, P.-A. Heng, and J. Wang, “Discrete wavelet transform on consumer-level graphics hardware,” IEEE Trans. Multime- it consumes very little power, especially in mobile GPUs. dia, vol. 9, no. 3, pp. 668–673, Apr. 2007. [26] C. Tenllado, J. Setoain, M. Prieto, L. Pinuel, and F. Tirado, “Parallel implementation of the 2D discrete wavelet transform on graphics process- REFERENCES ing units: Filter bank versus lifting,” IEEE Trans. Parallel Distrib. Syst., [1] Digital compression and coding for continuous-tone still images, ISO/IEC vol. 19, no. 3, pp. 299–310, Mar. 2008. Std. 10 918-1, 1992. [27] J. Matela, “GPU-Based DWT acceleration for JPEG2000,” in In Annual [2] N. Ahmed, T. Natarajan, and K. R. Rao, “Discrete cosine transform,” IEEE Doctoral Workshop on Mathematical and Engineering Methods in Com- Trans. Comput., vol. C-23, no. 1, pp. 90–93, Jan. 1974. puter Science, Nov. 2009, pp. 136–143. [3] D. Huffman, “A method for the construction of minimum redundancy [28] J. Franco, G. Bernabe, J. Fernandez, and M. E. Acacio, “A parallel codes,” Proc. IRE, vol. 40, pp. 1098–1101, 1952. implementation of the 2D wavelet transform using CUDA,” in Proc. [4] Information technology - JPEG 2000 image coding system - Part 1: Core IEEE International Conference on Parallel, Distributed and Network-based coding system, ISO/IEC Std. 15 444-1, Dec. 2000. Processing, Feb. 2009, pp. 111–118. [5] I. Daubechies, Ten lectures on wavelets. Philadelphia, PA: SIAM, 1992. [29] W. J. van der Laan, A. C. Jalba, and J. B. Roerdink, “Accelerating wavelet [6] F. Auli-Llinas and M. W. Marcellin, “Scanning order strategies for bitplane lifting on graphics hardware using CUDA,” IEEE Trans. Parallel Distrib. image coding,” IEEE Trans. Image Process., vol. 21, no. 4, pp. 1920–1933, Syst., vol. 22, no. 1, pp. 132–146, Jan. 2011. Apr. 2012. [30] V. Galiano, O. Lopez, M. P. Malumbres, and H. Migallon, “Speeding- [7] High Efficiency Video Coding Standard, International Telecommunication up the discrete wavelet transform computation with multicore and GPU- Union Std. H.265, 2013. based algorithms,” in Proc. International Conference on Computational [8] D. S. Taubman and M. W. Marcellin, JPEG2000 Image compression and Mathematical Methods in Science and Engineering, Jan. 2012, pp. fundamentals, standards and practice. Norwell, Massachusetts 02061 151–158. USA: Kluwer Academic Publishers, 2002. [31] ——, “Parallel strategies for 2D discrete wavelet transform in shared [9] A. Descampe, F.-O. Devaux, G. Rouvroy, J.-D. Legat, J.-J. Quisquater, and memory systems and GPUs,” SPRINGER The Journal of Supercomputing, B. Macq, “A flexible hardware JPEG 2000 decoder for digital cinema,” vol. 64, no. 1, pp. 4–16, Mar. 2012. IEEE Trans. Circuits Syst. Video Technol., vol. 16, no. 11, pp. 1397–1410, [32] P. Enfedaque, F. Auli-Llinas, and J. C. Moure, “Implementation of the Nov. 2006. DWT in a GPU through a register-based strategy,” IEEE Trans. Parallel [10] T. Bruylants, A. Munteanu, and P. Schelkens, “Wavelet based volumetric Distrib. Syst., vol. 26, no. 12, pp. 3394–3406, Dec. 2015. medical image compression,” ELSEVIER Signal Processing: Image Com- [33] T. M. Quan and W.-K. Jeong, “A fast discrete wavelet transform using munication, vol. 31, no. C, pp. 112–133, Feb. 2015. hybrid parallelism on GPUs,” IEEE Trans. Parallel Distrib. Syst., vol. 27, [11] B. Penna, T. Tillo, E. Magli, and G. Olmo, “Transform coding techniques no. 11, pp. 3088–3100, Nov. 2016. for lossy hyperspectral data compression,” IEEE Trans. Geosci. Remote [34] S. Datla and N. S. Gidijala, “Parallelizing motion JPEG 2000 with CUDA,” Sens., vol. 45, no. 5, pp. 1408–1421, May 2007. in Proc. IEEE International Conference on Computer and Electrical Engi- [12] Q. Huang, R. Zhou, and Z. Hong, “Low memory and low complexity VLSI neering, Dec. 2009, pp. 630–634. VOLUME VOL, YEAR 13 Carlos de Cea et al.: GPU-oriented architecture for an end-to-end image/video codec based on JPEG2000 [35] R. Le, I. R. Bahar, and J. L. Mundy, “A novel parallel tier-1 coder [49] C. de Cea-Dominguez, J. C. Moure, J. Bartrina-Rapesta, and F. Auli- for JPEG2000 using GPUs,” in Proc. IEEE Symposium on Application Llinas, “GPU architecture for wavelet-based video coding acceleration,” Specific Processors, Jun. 2011, pp. 129–136. in Parallel Computing: Technology Trends, vol. 36, Apr. 2020, pp. 83–92, [36] J. Matela, V. Rusnak, and P. Holub, “Efficient JPEG2000 EBCOT context IOSPress Series in Advances in Parallel Computing. modeling for massively parallel architectures,” in Proc. IEEE Data Com- [50] Nvidia, “Warp level primitives,” Tech. Rep., Jan. 2018. [Online]. pression Conference, Mar. 2011, pp. 423–432. Available: https://devblogs.nvidia.com/using-cuda-warp-level-primitives/ [37] F. Wei, Q. Cui, and Y. Li, “Fine-granular parallel EBCOT and optimization [51] ——, “Nvidia Tesla V100 GPU architecture,” Tech. Rep., Jun. 2019. with CUDA for digital cinema image compression,” in Proc. IEEE Inter- [Online]. Available: http://images.nvidia.com/content/volta-architecture/ national Conference on Multimedia and Expo, Jul. 2012, pp. 1051–1054. pdf/volta-architecture-whitepaper.pdf [38] M. Ciznicki, K. Kurowski, and A. Plaza, “Graphics processing unit im- [52] F. N. Iandola, D. Sheffield, M. Anderson, P. M. Phothilimthana, and plementation of JPEG2000 for hyperspectral image compression,” SPIE K. Keutzer, “Communication-minimizing 2d convolution in gpu registers,” Journal of Applied Remote Sensing, vol. 6, pp. 1–14, Jan. 2012. in Proc. IEEE International Conference on Image Processing, Sep. 2013, [39] J. Lee, B. Kim, and K. Yoon, “CUDA-based JPEG2000 encoding scheme,” pp. 2116–2120. in Proc. IEEE International Conference on Advanced Communication [53] A. Chacon, S. Marco-Sola, A. Espinosa, P. Ribeca, and J. C. Moure, Technology, Feb. 2014, pp. 671–674. “Thread-cooperative, bit-parallel computation of Levenshtein distance on [40] X. Wu, Y. Li, K. Liu, K. Wang, and L. Wang, “Massive parallel imple- GPU,” in Proc. ACM International Conference on Supercomputing, Jun. mentation of JPEG2000 decoding algorithm with multi-GPUs,” in Proc. 2014, pp. 103–112. SPIE Satellite Data Compression, Communications, and Processing X, [54] ——, “FM-index on GPU: a cooperative scheme to reduce memory foot- vol. 9124, May 2014, pp. 1–6. print,” in Proc. IEEE International Symposium on Parallel and Distributed [41] F. Auli-Llinas and M. W. Marcellin, “Stationary probability model for Processing with Applications, Aug. 2014, pp. 1–9. microscopic parallelism in JPEG2000,” IEEE Trans. Multimedia, vol. 16, [55] W. Sweldens, “The lifting scheme: A construction of second generation no. 4, pp. 960–970, Jun. 2014. wavelets,” SIAM Journal on Mathematical Analysis, vol. 29, no. 2, pp. [42] F. Auli-Llinas, “Context-adaptive binary arithmetic coding with fixed- 511–546, 1998. length codewords,” IEEE Trans. Multimedia, vol. 17, no. 8, pp. 1385– [56] F. Auli-Llinas and J. Serra-Sagrista, “JPEG2000 quality scalability without 1390, Aug. 2015. quality layers,” IEEE Trans. Circuits Syst. Video Technol., vol. 18, no. 7, [43] F. Auli-Llinas, P. Enfedaque, J. C. Moure, I. Blanes, and V. Sanchez, pp. 923–936, Jul. 2008. “Strategy of microscopic parallelism for bitplane image coding,” in Proc. [57] Nvidia. (2018, Dec.) CUB framework. [Online]. Available: https: IEEE Data Compression Conference, Apr. 2015, pp. 163–172. //nvlabs.github.io/cub/ [44] F. Auli-Llinas, P. Enfedaque, J. C. Moure, and V. Sanchez, “Bitplane image [58] ——, “Jetson SDK,” Tech. Rep., May 2019. [Online]. Available: coding with parallel coefficient processing,” IEEE Trans. Image Process., https://developer.nvidia.com/embedded-computing vol. 25, no. 1, pp. 209–219, Jan. 2016. [59] D. Taubman. (2020, Feb.) Kakadu software. [Online]. Available: [45] P. Enfedaque, F. Auli-Llinas, and J. C. Moure, “GPU implementation of http://www.kakadusoftware.com bitplane coding with parallel coefficient processing for high performance [60] University of Stuttgart. (2020, Jan.) CuJ2K. [Online]. Available: image compression,” IEEE Trans. Parallel Distrib. Syst., vol. 28, no. 8, pp. http://cuj2k.sourceforge.net/ 2272–2284, Aug. 2017. [61] Poznan Supercomputing and Networking Center. (2020, Feb.) GPUJ2K. [46] D. Taubman, A. Naman, and R. Mathew, “High throughput block coding in [Online]. Available: http://apps.man.poznan.pl/trac/jpeg2k/wiki the HTJ2K compression standard,” in Proc. IEEE International Conference [62] Comprimato. (2020, Feb.) Comprimato JPEG2000@GPU. [Online]. on Image Processing, Sep. 2019, pp. 1079–1083. Available: http://www.comprimato.com [47] A. Naman and D. Taubman, “Decoding high-throughput JPEG2000 [63] Fastvideo LLC. (2020, Feb.) CUDA-JPEG2K. [Online]. Available: (HTJ2K) on a GPU,” in Proc. IEEE International Conference on Image https://www.fastcompression.com/products/gpu-jpeg2000.htm Processing, Sep. 2019, pp. 1084–1088. [64] Nvidia. (2018, Dec.) HEVC SDK. [Online]. Available: https://developer. [48] C. de Cea-Dominguez, P. Enfedaque, J. C. Moure, J. Bartrina-Rapesta, and nvidia.com/nvidia-video-codec-sdk F. Auli-Llinas, “High throughput image codec for high-resolution satellite images,” in Proc. IEEE International Geoscience and Remote Sensing Symposium, Jul. 2018, pp. 6524–6527. 14 VOLUME VOL, YEAR