Received March 10, 2020, accepted March 25, 2020, date of publication April 6, 2020, date of current version April 22, 2020. Digital Object Identifier 10.1109/ACCESS.2020.2985859 GPU-Oriented Architecture for an End-to-End Image/Video Codec Based on JPEG2000 CARLOS DE CEA-DOMINGUEZ 1, JUAN C. MOURE-LOPEZ 2, JOAN BARTRINA-RAPESTA 1, AND FRANCESC AULÍ-LLINÀS 1, (Senior Member, IEEE) 1 Department of Information and Communications Engineering, Universitat Autònoma de Barcelona, 08193 Barcelona, Spain 2 Department of Computer Architecture and Operating Systems, Universitat Autònoma de Barcelona, 08193 Barcelona, Spain Corresponding author: Carlos de Cea-Dominguez (carlos.decea@uab.cat) This work was supported in part by the Spanish Ministry of Economy and Competitiveness and the European Regional Development Fund under Grant TIN2017-84553-C2-1-R and Grant RTI2018-095287-B-I00 (MINECO/FEDER, UE), and in part by the Catalan Government under Grant 2017SGR-463 and Grant 2017SGR-313. ABSTRACT Modern image and video compression standards employ computationally intensive algorithms that provide advanced features to the coding system. Current standards often need to be implemented in hard- ware or using expensive solutions to meet the real-time requirements of some environments. Contrarily to this trend, this paper proposes an end-to-end codec architecture running on inexpensive Graphics Processing Units (GPUs) that is based on, though not compatible with, the JPEG2000 international standard for image and video compression. When executed in a commodity Nvidia GPU, it achieves real time processing of 12K video. The proposed S/W architecture utilizes four CUDA kernels that minimize memory transfers, use registers instead of shared memory, and employ a double-buffer strategy to optimize the streaming of data. The analysis of throughput indicates that the proposed codec yields results at least 10× superior on average to those achieved with JPEG2000 implementations devised for CPUs, and approximately 4× superior to those achieved with hardwired solutions of the HEVC/H.265 video compression standard. INDEX TERMS Wavelet-based image coding, high-throughput image coding, JPEG2000, GPU, CUDA. I. INTRODUCTION samples1 of the image multiple times, consuming significant Over the past decades, the computational complexity of processing time even when executed in the latest processors. image and video coding systems has increased notably. JPEG2000 is a widespread standard in fields that deal with In the early nineties, the JPEG standard (ISO/IEC 10918) [1] large sets of images and/or videos. Its coding pipeline has employed the low-complexity discrete cosine transform [2] three main stages [8]. The first reduces the image redundancy and Huffman [3] coding. Ten years after, the JPEG2000 stan- through a color transform (CT) and the DWT. The second dard (ISO/IEC 15444) [4] introduced more computationally employs bitplane coding together with arithmetic coding to demanding algorithms such as the discrete wavelet trans- reduce the statistical redundancy of wavelet coefficients. The form (DWT) [5] and bitplane coding [6]. In the last years, third reorganizes the data to produce the final codestream. HEVC/H.265 (ISO/IEC 23008) [7] doubled the compression The high computational complexity of these stages poses a efficiency of previous standards by using complex techniques challenge to meet the real-time requirements of some sce- that exploit intra- and inter-redundancy of frames. Nowa- narios. In Digital Cinema, for instance, JPEG2000 needs days, most codecs (including JPEG2000 and HEVC) provide to be implemented in Field-Programmable Gate Arrays to advanced features such as scalability by quality, interac- process 2K (i.e., 2048 × 1024) and 4K (i.e., 4096 × 2048) tive transmission, and error resilience, among others. To do resolution [9]. In medical and remote sensing applications, so, they use algorithms that scan, transform, and code the dedicated servers and workstations are employed to manage and store the large quantity of images that are produced 1 A sample is the basic unit of a digital image, representing a level of The associate editor coordinating the review of this manuscript and brightness in a grayscale or color component (each RGB pixel has three approving it for publication was Zhaoqing Pan . samples). This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ 68474 VOLUME 8, 2020 C. de Cea-Dominguez et al.: GPU-Oriented Architecture for an End-to-End Image/Video Codec Based on JPEG2000 daily [10], [11]. This has motivated many works in the litera- 15) adopts the algorithm proposed in [46]. Such algorithm ture that propose hardware architectures to accelerate partic- is devised to mostly benefit from the modern instruction ular stages of the JPEG2000 coding pipeline [12]–[22]. sets like AVX2, NEON, and BMI2 included in new CPUs, Highly parallel architectures may help to reduce process- though it can also be implemented in GPUs [47]. It is about ing time and costs in some environments. Graphics Process- 10× faster than the standard, but it penalizes coding per- ing Units (GPUs) may be ideal due to their high throughput, formance in approximately 10%. Also, it sacrifices quality low cost, and widespread availability. Their architecture is scalability, which is a valued feature of the system since mainly based on the Single Instruction Multiple Data (SIMD) it permits the transmission of an image progressively by paradigm, which executes a flow of instructions on mul- quality. tiple data in a lock-step synchronous way. When the pro- This paper introduces a highly-parallel, GPU-oriented gram allows data (in addition to task) parallelism, thou- codec based on JPEG2000. The proposed codec is the final sands of threads can be executed in parallel, achieving a piece of our research line that was aimed to explore new cod- throughput that is potentially an order of magnitude higher ing techniques for image/video compression tailored for the than that achieved by conventional Central Processing Units fine-grain parallelism of GPUs. The JPEG2000 framework is (CPUs) [23]. This is in part because the architecture of the employed to show that the proposed techniques can virtually CPUs is more based on the Multiple Instruction Multiple obtain the same coding performance of this standard without Data (MIMD) paradigm, which allows the asynchronous exe- sacrificing any feature. Evidently, compliance with the stan- cution of fewer threads over different sets of data. dard is lost since the proposed techniques require significant Most of the workload in the first stage of the changes in the core coding system. A preliminary version JPEG2000 pipeline lies in the DWT, which is well-suited of the proposed codec was partially described in [48], [49]. to the SIMD paradigm. The first implementations of the This paper vastly improves our previous work by describing DWT for GPUs appeared in the 2000s making use of the the complete coding pipeline with the needed machinery graphics pipeline [24]–[26]. Later, the use of the Compute to avoid bottlenecks, providing the color transform and the Unified Device Architecture (CUDA) programming language codestream reorganization stages with an in-depth analysis introduced by Nvidia increased the throughput of such imple- of the kernel metrics and memory transfers, and report- mentations significantly [27]–[31]. Recently, we proposed a ing extensive experimental tests. The obtained results show register-based implementation of the DWT for GPUs [32] that the proposed S/W architecture can process real-time that yields 40× speedups compared to CPU implementations. 12K (i.e., 12288 × 6144) video, achieving a throughput Similar results are also achieved in [33]. 4× superior to that achieved by the state-of-the-art Nvidia In general, the DWT takes 15% of the total execution time codec of HEVC that is supported by in-chip dedicated of the codec. The most expensive stage is the bitplane and hardware. arithmetic coding, which spends about 80% of the time. This The rest of the paper is structured as follows. Section II stage poses the major challenge for GPUs because it is not briefly overviews the architecture of Nvidia GPUs and well-suited to the SIMD paradigm. In this stage, the wavelet- JPEG2000. Section III describes the proposed codec from transformed image is partitioned in small sets of typically a top-down perspective and Section IV details each kernel 64 × 64 wavelet coefficients, called codeblocks, and codes employed. Section V evaluates the throughput of our archi- them independently. This provides coarse-grain parallelism. tecture and compares it to some of the fastest JPEG2000 The coding within each codeblock must be carried out by a and HEVC implementations. The last section contains single thread, since there exist causal relationships among conclusions. coefficients. This means that the coding of a coefficient depends on the output of the previous, so they can not be II. BACKGROUND processed in parallel. Even so, there have been efforts to A. NVIDIA GPU ARCHITECTURE implement this stage in GPUs [34]–[40], though these solu- Nvidia GPUs are hardware devices that are mainly consti- tions do not to fully occupy the resources of the GPU due tuted by individual computing units called Streaming Mul- to the lack of fine-grain parallelism. In 2014, we started a tiprocessors (SMs). Depending on the model and the archi- line of research [41]–[45] focused on providing fine-grain tecture, a Nvidia GPU may contain from one to tens of parallelism to this stage without sacrificing any feature of SMs. Each SM can work independently, allowing the GPU to the system. The goal was not to implement the compliant process sequences of instructions from different algorithms. JPEG2000 algorithm, but to redevise it keeping in mind the Typically, SMs execute multiple 32-wide vector instructions SIMD architecture of GPUs. The proposed algorithm is not in parallel. compatible with the standard, but it allows parallel coefficient CUDA refers vector instructions as warps. Each lane of processing within the codeblock. a vector is virtualized into a software thread. Aggregations Following a similar line, in 2017 the Joint Photographics of 32 threads form a warp. A group of warps, called thread Experts Group launched a call for proposals with the aim block, is assigned to a SM for execution. From the first to augment the parallelism in the second stage of the cod- CUDA-compatible architecture (v1.0) up to Pascal (v6.2), ing pipeline. This new part of JPEG2000 (ISO/IEC 15444- warps are always executed synchronously and in a lock-step VOLUME 8, 2020 68475 C. de Cea-Dominguez et al.: GPU-Oriented Architecture for an End-to-End Image/Video Codec Based on JPEG2000 FIGURE 1. JPEG2000 coding pipeline. fashion, featuring an implicit synchronization at the end of B. JPEG2000 any divergence [50]. Volta (v7.0) introduced a modification in As previously stated, JPEG2000 is an image/video coding the warp scheduler that allows the execution of warp threads standard employed in professional environments due to its asynchronously [51], so the synchronization among threads excellent features and performance. The proposed codec car- must be explicitly programmed when needed. Our codec is ries out almost the same operations as JPEG2000, so they are adapted to work with both implicit and explicit synchroniza- briefly described herein for completeness. Figure 1 depicts tion. these operations. Depending on whether lossy or lossless The memory architecture of the GPU is organized in three compression is needed, some of these operations are irre- levels: global, shared, and local. The size of the global mem- versible or reversible. As stated before, the first stage of ory is, in general, in the order of GBs and is accessible by JPEG2000 applies several transformations to the image. The all SMs. When this memory is accessed in a coalesced way first is carried out for color images, converting the red, green, (i.e., via consecutive positions) the available bandwidth is and blue (RGB) components to the lesser redundant color used efficiently and the latency is minimized. The size of the space YCb Cr , which holds the luminance information in the shared memory is in the order of MBs and its latency is lower first component and the chrominance with respect to blue and than that of the global, though it can only be shared within the red in the second and third components, respectively. This thread blocks. The local memory is the fastest though it is also is a pixel-wise operation that holds no dependencies among limited in size and is only accessible by the threads within a pixels. It is carried out applying floating-point or integer warp. The data allocated in the local memory are commonly operations for the irreversible or reversible path, respectively. stored in the registers, though they may be temporarily moved The second operation is the DWT. Most implementations to the device memory (i.e., DRAM of the GPU) when the apply it via the lifting scheme [55] since it has low com- register space is saturated. Typically, the global memory is putational complexity. The main idea behind this scheme is employed to read and store the application’s data, the shared to first apply a series of arithmetic operations to all rows of memory is used for communication among threads of differ- the image and then to all columns. These operations can be ent warps, and the local memory is utilized for intermediate carried out in parallel to all rows and then to all columns since computation. The local memory can be shared among threads there are no inter-row/column dependencies. Then, the result- within a warp via the low-level shuffle operation. This kind ing coefficients are re-ordered taking the coefficients in the of memory sharing technique proved to be very efficient in even and odd positions in each direction. This produces four some applications [32], [52]–[54]. The GPU has two levels of different subbands of one quarter the size of the original cache, denoted by L1 and L2. The registers and the L1 cache image. In general, the same procedure is applied again four are in the SM. The data transferred from the device memory to more times in the subband that contains the low-detail image. the registers passes through the L1 and L2 caches, which are The operations carried out in each step apply a low- and reservoirs of the most recently accessed data to be (possibly) high-pass filter. JPEG2000 uses the irreversible CDF 9/7 and reused in future petitions. the reversible CDF 5/3. As previously mentioned, each SM runs thread blocks. The irreversible filter bank employs floating-point arith- These blocks can execute code from one or more CUDA metic, so the resulting coefficients need to be converted to functions, called kernels, independently. This allows the par- integers before bitplane coding. This operation is called dead- allel execution of many different kernels from a single or zone quantization [8]. It multiplies the coefficients by a step various applications. CUDA provides the so-called streams to size and keeps the integer part. This operation is not necessary organize the execution of running kernels. Each stream may for the reversible transform since it already produces integer process a sequence of kernels of an application in a set of coefficients. SMs asynchronously from the rest. An appropriate use of the The second main stage of the coding pipeline carries out streams optimizes the use of the GPU resources, which can bitplane coding together with arithmetic coding. As stated help to increase the throughput. before, this stage is applied in each codeblock independently. 68476 VOLUME 8, 2020 C. de Cea-Dominguez et al.: GPU-Oriented Architecture for an End-to-End Image/Video Codec Based on JPEG2000 FIGURE 2. Illustration of the codec architecture when using 2 CUDA streams. The cycle of the data is as follows. First, frame data (individually identified by color) are read from disk to a RAM buffer. Then the data are managed by a stream in the GPU. Within the device the data are transferred from global memory MD to local memory R and inversely before and after running each kernel. The kernel execution is illustrated by the matrix of 0s and 1s. Each stream processes the three components of the frame before transferring the compressed data back to the host memory MH and disk. Through the binary representation of the integer coefficients re-organizes these bitstreams to put them in the compressed (without sign), a bitplane is defined as the set of bits from file together with ancillary information for decoding. The all coefficients in the same binary position. Bitplanes are decoder carries out the same operations in reverse order coded from the most to the least significant. Just after the except the rate-distortion optimization stage, which is not first non-zero bit of a coefficient is coded (referred to as necessary. significance bit), its sign is coded too so that the decoder can reconstruct that coefficient. The bits coded for a coeffi- III. OVERVIEW OF THE CODEC ARCHITECTURE cient after its significance bit are called refinement bits. The A. OVERVIEW coefficients within a codeblock are scanned in a pre-defined Except for bitplane and arithmetic coding, all operations of order that visits four rows of coefficients, called stripes, the JPEG2000 coding pipeline offer fine-grain parallelism. consecutively. In each stripe, coefficients are scanned from Our codec implements these operations following the stan- the left- to right-most column and, in each column, from the dard, so their input/output is the same as that obtained top to the bottom row. JPEG2000 codes each bitplane in three by a conventional JPEG2000 implementation. To use the coding passes. The first is called significance propagation. JPEG2000’s bitplane and arithmetic coder would signifi- It follows the scanning order processing only those coeffi- cantly hinder the throughput of the GPU, so this is the only cients that have at least one significant neighbor. The second stage that is not compliant with the standard. This stage is is called magnitude refinement. It processes coefficients that replaced by the coding engine proposed in [44], [45]. The aim were found significant in previous bitplanes. The third pass of our codec is to code large quantities of images. The input processes the remaining coefficients. It is called cleanup. This data set may contain frames of a video sequence or images multiple-pass coding is aimed to code first the information of the same size. For convenience, frame is used to refer both that reduces the most the distortion of the image [6]. terms in the following. Each processed bit is fed to the arithmetic coder together When possible, the proposed architecture joins operations with its contextual information. The context considers the sig- in a single kernel instead of using a straightforward approach nificance, or sign, of its eight neighbors. One of 18 different that uses one kernel per operation. Within the same kernel, pre-defined contexts is chosen depending on this information. the data are always accessed in the same fashion and the data The context of the coefficient is employed by the arithmetic types do not change. This permits the kernels to maximize the coder to establish a probability for the currently processed bit, use of local memory in detriment of shared memory, using generating a compacted stream of bits. a register-based strategy [52]–[54] that minimizes memory The output produced in this stage for each codeblock is a latencies. When the data set needs to be re-organized or the bitstream that can be truncated at the end of each coding pass. data type is changed, then the data are transferred to the global Like most coding systems, JPEG2000 permits specifying a memory preparing them for the next kernel. This architecture size for the final codestream, so bitstreams may be truncated minimizes the overall memory transfers and significantly to fit the target rate. This rate-distortion optimization proce- increases performance. dure is not defined in the standard, so each codec can choose Algorithm 1 describes the main routine of the codec. Its among a great variety of methods [56]. The final operation architecture is also illustrated in Figure 2. First, all memory VOLUME 8, 2020 68477 C. de Cea-Dominguez et al.: GPU-Oriented Architecture for an End-to-End Image/Video Codec Based on JPEG2000 TABLE 1. Evaluation of the memory bandwidth achieved by our codec when transferring data from host to device (MH → MD ) and device to host (MD → MH ) with pinned and paged memory, for two different GPUs. Algorithm 1 Main Routine of the Codec each frame. The reading of frames is managed by a thread, 1: CPUMemoryAllocation() denoted by t1 in Algorithm 1, that is executed by the host. 2: GPUGlobalMemoryAllocation() Each stream, denoted by Sj , j ∈ {1..b S} with bS being the  3: for each empty MH [i] do number of streams, employs two input buffers in both MH and MD so that when a buffer is being processed the other  MH [i] ← HDRead()  4: t1 can be filled. These buffers are referred to as MH [i], MD [i]  5:  MD [i] ← MH [i] 6: end for with i ∈ {1..2b S}. This filling is carried out in lines 3-6. t1  7: A D 1..3 ← CT(M [i]) continuously checks if there is any empty buffer in MH .     8: for k ∈ {1..3} do If so, it reads the data from disk and transfers them to MH .   9: Dk ← DWT_Q(Ak ) Then, it issues an asynchronous copy to the device memory Sj in line 5. t1 is active until all frames have been buffered. The  10: {Bl } ← BPC_AC(Dk )     11: MD [o] ← CR({Bl }) data are read and stored considering their original bit-depth  12: end for to optimize transfers and memory space. In general, 8-bit  13: for each filled MD [o] do integers are employed.   14: MH [o] ← MD [o] The writing of the compressed data to the disk is done t2 15: HDWrite(MH [o]) similarly by thread t2 , which is executed by the host in   16: end for lines 13-16. A double-buffer strategy is also employed so that when a stream finishes coding a frame, it can readily start coding another without waiting for the compressed data needed during the coding process is pre-allocated both in the to be transferred to the host memory. These output buffers host RAM and the device DRAM, which are respectively are referred to as MH [o], MD [o] with o ∈ {1..2b S}. Again, referred to as MH and MD . This allocation (lines 1 and 2 in the data transfer from device to host is carried out via an the algorithm) considers the space needed for a double buffer asynchronous copy in line 14. Once the transfer is done, strategy to load the frames (see below), auxiliary memory t2 writes them to the disk. The data are copied in the disk structures, and number of GPU streams employed. The host orderly, i.e., following the same frame order of the original RAM allocation is performed in pinned memory2 to avoid sequence. memory positions requests to the CPU when transferring Lines 7-12 in Algorithm 1 describe the calls to the ker- data. This allocation greatly improves the memory bandwidth nels and the auxiliary memory structures employed in the achieved in some GPUs. See, for instance, in Table 1 the GPU. Four kernels are used. The first carries out the color difference in the bandwidth achieved by our codec when transform. It transfers all frame data from MD [i] to local coding a 4K video (with the test environment described in memory converting them to 32-bits integers (floats) for the Section V) using pinned or paged memory. To use pinned (ir)reversible path and performs the arithmetic operations memory in the Nvidia GTX 1080 Ti (Pascal architecture) on the registers. The result is left in the auxiliary struc- almost doubles the bandwidth achieved as compared to paged ture denoted by A1..3 using the same data type employed memory. For the RTX 2080 Ti (Turing architecture), the dif- in the kernel. After this, each component is processed ferences are much smaller due to the use of DDR4 RAM independently. The next kernel carries out the DWT and, modules in the host, though there is a slight increase of 4% in if using lossy compression, quantization. Our codec employs the bandwidth achieved. It is worth noting that the practical a rate-distortion optimization method that controls the rate maximum speed of the PCI-E 3.0 bus employed is 13.2 GB/s through the quantization step employed in this operation [56]. (with 15.8 GB/s of theoretical maximum), so our codec yields It transfers the data from Ak to the registers, applies the maximum bandwidth in practice. lifting scheme, and leaves the result in Dk . The third kernel is Memory transfers are programmed to be asynchronous the most complex. It applies bitplane and arithmetic coding. so they can absorb variations in the time spent to process Like the other kernels, it reads the data from the global memory and puts them in the local. These data are organized 2 Pinned memory indicates that the allocated space has a fixed location in in codeblocks holding 64×64 coefficients. Each codeblock is the RAM module(s) during the whole execution. processed by an individual warp of 32 threads. The result of 68478 VOLUME 8, 2020 C. de Cea-Dominguez et al.: GPU-Oriented Architecture for an End-to-End Image/Video Codec Based on JPEG2000 TABLE 2. Analysis of the codec’s kernels when coding a 2K and 4K frame with the Nvidia RTX 2080 Ti. TABLE 3. Analysis of the hierarchical memory transfers of the codec’s kernels when coding a 2K and 4K frame with the Nvidia RTX 2080 Ti. this kernel is stored in the set {Bl } that contains one bitstream when the data are transferred back from the registers to the per codeblock, with l ∈ {1..b L} and b L being the number of device memory via the L1 and L2 caches. codeblocks per component. The length of each bitstream is The DWT_Q(·) kernel can perform a variable number not known before coding, so the space for bitstreams {Bl } of transformation levels, typically 5. The metrics reported is pre-allocated amply. As a result, the bitstream data are in Table 2 correspond to the first call to the kernel, which scattered throughout the whole structure. These data must be performs the first level of transformation. The achieved occu- compacted before transferring them to the host memory and pancy is about 84% for 2K and 90% for 4K. This indicates disk, which is the function of the last kernel. Contrarily to the that other computations can be done while this kernel is other kernels, it does not put the frame data to the registers but running. Similar to the previous kernel, the warp efficiency is only the lengths of the generated bitstreams (via pointers to almost 100% since there are no divergent paths. The increase memory positions), so that it can compute the final position of in execution time and total number of instructions between each compressed byte. Then, it re-organizes the compressed 2K and 4K is also proportional to the frame size. As seen frame data in the global memory leaving them in one of the in Table 3, this kernel utilizes more registers per thread due two output buffers. to a larger data tile processed by each warp. The data require The decoder employs a similar structure to that of the 8 MB and 32 MB for the 2K and 4K frame, respectively, encoder. It executes the kernels in inverse order, performing which approximately correspond to the transfers between the reverse operations. MD to R and inversely. The extra data transferred correspond to auxiliary information. The transfers between the L1 and B. ANALYSIS L2 cache are higher than those from the device memory Table 2 and 3 report the kernels’ metrics obtained via the to the registers because this kernel processes the data tiles Nvidia Nsight Compute tool when coding a 2K and 4K frame employing a redundant halo. using the test environment described in Section V. The first As shown by the metrics, the BPC_AC(·) kernel is the kernel (i.e, ICT(·)) achieves high occupancy, optimal warp most complex. First, the occupancy is much lower than that efficiency (since it does not have divergence), and very high achieved by the other kernels, especially for 2K frames. This memory bandwidth (see Table 2). These results are due to is because 2K frames do not have enough data to fill the the pixel-wise operation that it carries out. The differences resources of the GPU. 4K frames achieve higher occupancy, between the 2K and 4K frame with respect to execution though it is still below that achieved by the other kernels. time and total number of instructions executed are a 4 fold Second, the warp efficiency is 63% due to the multiple diver- increase, coinciding with the increase in number of processed gent paths of the algorithm. Third, the memory bandwidth samples. We recall that this kernel processes the three image is much lower than that achieved by the other kernels since components, whereas the following kernels process only one. BPC_AC(·) is bounded by the latency of the computing As seen in Table 3, the three image components are trans- instructions [45]. Fourth, the time spent for coding a 2K ferred from MD to R requiring 6 and 24 MB for a 2K and and 4K frame is not proportional to the frame size. This is 4K frame, respectively. Once the data are in the SM, they due to the low occupancy that is achieved for 2K frames are converted from 8-bit integers to 32-bit integers or floats and due to the image content. Let us explain further. The depending on whether the reversible or irreversible transform codeblock size is 64 × 64 regardless of the frame size. This is selected. This conversion is seen in the memory transfers causes that codeblocks of 2K frames have more details (i.e., VOLUME 8, 2020 68479 C. de Cea-Dominguez et al.: GPU-Oriented Architecture for an End-to-End Image/Video Codec Based on JPEG2000 more entropy) than codeblocks of 4K frames, requiring more instructions to code their information. This is manifested in the total number of instructions and instructions per sample executed, since the 4K frame requires approximately 20% fewer instructions to code each sample. The memory transfers when reading the data are higher than in the other kernels mainly due to the register pool size (see Table 3). Differently from the previous kernels, BPC_AC(·) visits each coefficient of the codeblock many times. The number of visits depends on the codeblock’s data, but is approximately 8 or 9 times per coefficient on average. Since the size of the register space is limited, once a coefficient is visited it is transferred back to the device memory so the register can be employed for other coefficients. When the coefficient is needed again, it is transferred from the device memory to the registers. Many FIGURE 3. Analysis of the throughput achieved by the proposed codec of these coefficients are kept in cache and are reused, so the when encoding 2K and 4K video using different number of execution streams, for the RTX 2080 Ti. transfers between the L2 and L1 cache are high as well. The data transfers when writing are not as high because the kernel only stores the compressed data. Even so, the data in the compressed bitstream are accessed many times, so the Algorithm 2 Kernel Routine CT(MD [i]) transfers between registers and device memory are higher 1: GPULocalMemoryAllocation() than in the previous kernels. 2: R1..3 ← MD [i] The occupancy and efficiency of the CR(·) kernel is similar 3: R01..3 ← φ(R1..3 ) to that achieved by ICT(·) and DWT_Q(·). The execution time 4: A1..3 ← R01..3 for 4K frames is twice as that needed for 2K. This is because 5: return(A1..3 ) both frames require 5µs to generate preliminary tables, and then the data to be reorganized are about 1 MB and 3 MB respectively for the 2K and 4K frame,3 requiring 10µs and the decoder requires more local memory, which reduces the 30µs. The memory bandwidth is lower than that obtained occupancy. The rest of the decoding process is very simi- in the first two kernels since the transferred data are already lar to that of the encoder, so it is not reported herein for compressed(also seen in Table 3). brevity. This analysis indicates that the BPC_AC(·) kernel con- sumes most of the total execution time and it achieves the IV. DESCRIPTION OF THE KERNELS lowest occupancy. This suggests that the codec may underuse Algorithm 2 details the routine of the CT(·) kernel. In this the resources of the GPU when coding large sets of images and following kernels, the algorithm describes the main oper- or video unless more workload is feed to the device. The pro- ations that are performed at a thread level. Like in the other posed architecture alleviates this issue by employing multiple kernels, the first instruction allocates the local memory. All streams of execution. Each stream processes a frame, so more kernels only use registers since this increases the throughput. data are processed in parallel, employing more resources After allocating the required space, the data of the three and increasing the overall throughput. See in Figure 3 the frame components are transferred from the global memory throughput achieved by our multiple-streamed codec when to the register space, referred to as R for the input data. encoding 2K and 4K video in the same conditions as before. This is the only kernel that needs the three components of The results are reported as the number of Mega Samples the frame. It applies a transformation that involves several coded per second (MS/s). The figure depicts the throughput arithmetic operations, denoted by φ(·) in line 3, and the result needed to code 4K, 8K, and 12K video in real-time with is left in the output register space R0 . Then the data are straight horizontal lines for the convenience of the reader. returned to the global memory, ready to be fetched by the As seen in the figure, the throughput increases notably when next kernel. Both reading and writing in the global memory multiple streams are employed. In the case of 2K (4K) video, in this and following kernels is carried out in a coalesced way 13∼14 (7∼8) streams obtain maximum efficiency. Again, to maximize memory performance since the GPU stores data the coding of 4K video achieves higher throughput due to the blocks adjacent to that requested in the L2 cache for (possi- nature of the data. ble) future requests. Depending on whether lossy or lossless As seen in Section V the throughput achieved by the compression is selected, the operations and the data types decoder is only slightly lower than that of the encoder because employed in the registers are floating points or integers, respectively. 3 4K frames are compressed more efficiently than 2K frames, so they The second kernel is detailed in Algorithm 3. The wavelet generate fewer data per sample coded. transform is applied in blocks of 64×b Y samples that are 68480 VOLUME 8, 2020 C. de Cea-Dominguez et al.: GPU-Oriented Architecture for an End-to-End Image/Video Codec Based on JPEG2000 Algorithm 3 Kernel Routine DWT_Q(Ak ) The BPC_AC(·) kernel is detailed in Algorithm 4. It is 1: GPULocalMemoryAllocation() applied to all codeblocks of the component, though we recall 2: R ← Ak that the algorithm details the operations carried out at thread 3: for y ∈ {1..bY } do level. The kernel receives a frame component that is parti- 4: for x ∈ {0..1} do tioned in codeblocks of 64×b Y 0 coefficients, with typically 0 Y = 64. The data for the codeblock are implicitly transferred 5: R[y][x] ← ϕ(R[y][x]) b 6: end for to the local memory in line 2 of the algorithm. Then the 7: end for coefficients are coded from bitplane b B, which is a sufficient 8: for x ∈ {0..1} do number of magnitude bits to code all coefficients within the 9: for y ∈ {1..bY } do codeblock, to the lowest bitplane 0. This is performed in 10: R[y][x] ← ϕ(R[y][x]) the loop of line 3. Like in the previous kernel, each thread 11: end for processes two columns and each codeblock is processed 12: end for by a warp. Contrarily to JPEG2000, this kernel carries out 13: for y ∈ {1..bY } do 2 coding passes instead of 3 since virtually same compres- 14: for x ∈ {0..1} do sion efficiency is achieved [6], [44], [45] while increasing 15: / halo then if (y, x) ∈ throughput about 40%. The loop in lines 4-17 performs 16: R[y][x] ← R[y][x] · Q significance coding. It checks whether the coefficient was 17: Dk ← R[y][x] significant in previous bitplanes via the γ (·) function, which 18: end if returns the significance bitplane of the coefficient. If not, 19: end for significance coding is performed. First, context C of the 20: end for coefficient is determined via 8(·) and, through this context 21: return(Dk ) and the current bitplane, probability P for the coded bit is extracted from the lookup table LUTsig . This table contains pre-computed probabilities determined with a training set processed by a single warp.4 This allows communication of images. Then, the bit is coded via arithmetic coding. among threads without needing shared memory. The height of The procedure for AC(·) is not detailed in the algorithm the block is denoted by b Y . Each thread processes two columns for simplicity. It can be found in [45]. If the coefficient is of a block. The kernel applies a 2D high-pass/low-pass filter significant in the current bitplane (i.e., γ (R[y][x]) = b), to all samples. First, the filter ϕ(·) is applied horizontally its sign is coded in lines 11-13 with a similar procedure to (lines 3-7) and then vertically (lines 8-12). The filter consists that of significance coding. Refinement coding is carried out in a series of arithmetic operations that use the adjacent in lines 18-25. In this case, no context is employed. The samples to the processed coefficient, in which the result is return of the AC(·) function is the bitstream Bl that contains left. This type of operation does not require two register the compressed information. Each time that this function is spaces (for input and output) like in the previous kernel, but called, some data may be added to Bl . We note that Bl is in the only one that is referred to as R. When the thread needs data global memory. Each thread puts data in Bl asynchronously from other threads, it uses shuffle instructions (not shown in from the others ensuring mutual exclusion. This exclusion is Algorithm 3) since they have lower latency than using shared guaranteed considering the threads that need a new chunk of memory [32]. If more than one level of wavelet transform memory to write their information, assigning positions based is selected, the instructions from line 3 to 12 are repeated on the thread index within the warp. This kernel also stores each time over a quarter of the last data processed, which the length of Bl in a separate global memory region, denoted contains the results of the low-pass filter. This is carried out by L. calling the kernel again. It is not detailed in Algorithm 3 for The last kernel (i.e., CR(·)) is detailed in Algorithm 5. the sake of clarity. The final step in this routine is to transfer It receives the set of bitstreams {Bl }. As previously stated, its the data from the local space to the global memory. It is purpose is to reorganize the bitstream data in a compact struc- only done for those samples that do not belong to the halo.5 ture. To do so, blocks of 2 bytes are assigned to each thread in Before transferring the data, a quantization step size, denoted the warp to be written in the final memory positions. The first by Q in line 16, may be applied. Again, lossy and lossless step is to generate a memory map to know these positions. compression respectively requires the use of floating points This map is denoted as L0 and contains an aggregated list of and integers when applying ϕ(·). Quantization is only applied lengths, more precisely, L0 = {0, L1 , L1 +L2 , · · · , L1 +· · ·+ for lossy compression. Lb 0 L }. L is generated via the Device Scan primitive from the Nvidia CUB framework [57]. To accelerate the access to this 4 Note that these blocks are not the codeblocks utilized in BPC_AC(·), but map, a fast lookup table, denoted by LUTL0 , is created. This a tile of the original image. Although the partitioning is similar for paral- LUT is generated applying a binary search over L0 in which lelism purposes, the block transformed by DWT_Q(·) contains overlapped samples of adjacent blocks. each position represents some positions of the original map. 5 The halo is an area surrounding the processed samples that is employed Our experience indicates that speedups about 2× are achieved by the warp to obtain the correct result of the wavelet transform. by using such a strategy. These operations are carried out VOLUME 8, 2020 68481 C. de Cea-Dominguez et al.: GPU-Oriented Architecture for an End-to-End Image/Video Codec Based on JPEG2000 Algorithm 4 Kernel Routine BPC_AC(Dk ) V. EXPERIMENTAL RESULTS 1: GPULocalMemoryAllocation() The proposed codec is evaluated with four Nvidia GPUs, 2: R ← Dk namely, the RTX 2080 Ti, the GTX 1080 Ti, the Xavier, 3: for b ∈ {b B..0} do and the Tegra X2. These devices are commodity GPUs, with 4: for y ∈ {1..bY 0 } do prices ranging from 650¿ to 1350¿. Their specifications are 5: for x ∈ {0..1} do reported in Table 4. Both the RTX 2080 Ti and the GTX 6: if γ (R[y][x]) ≤ b then 1080 Ti are commonly employed in workstations for design 7: C ← 8(R[y][x]) applications and gaming. The RTX 2080 Ti has the highest 8: P ← LUTsig [C][b] peak throughput. It is employed with an i9 9900K CPU 9: Bl ← AC(R[y][x], P) workstation with 16 GB of DDR4 RAM. The 1080 Ti is used 10: if γ (R[y][x]) = b then on an i7-3770 workstation with 8 GB of DDR3 RAM. Both 11: C 0 ← 80 (R[y][x]) the Xavier and the Tegra X2 are GPUs devised for devices in 12: P0 ← LUTsign [C 0 ][b] which efficiency and size are important aspects, for example 13: Bl ← AC(R[y][x], P0 ) in the Nintendo Switch. In our tests, they run on a Jetson 14: end if SDK platform [58]. Both GPUs have low performance, but 15: end if consume very little power. Both allow different power modes 16: end for with varying performance and Thermal Design Power (TDP). 17: end for The results reported below correspond to the maximum per- 18: for y ∈ {1..bY 0 } do formance mode except when indicated. 19: for x ∈ {0..1} do JPEG2000 results are obtained with Kakadu (v8.0.2) [59]. 20: if γ (R[y][x]) > b then Kakadu is among the fastest CPU implementations of the 21: P00 ← LUTref [b] standard. It is heavily optimized in assembler, achieving 22: Bl ← AC(R [y][x], P00 ) superior throughput than other implementations for GPUs 23: end if such as CuJ2K [60] and GPU-J2K [61]. It is executed in 24: end for a workstation with an Intel i9-9900K CPU with 8 cores 25: end for and 16 GB of DDR4 RAM. Kakadu is compiled for this 26: end for architecture and it is run with 16 threads of execution to 27: Ll ← length(Bl ) achieve maximum throughput. The compression parameters 28: return(Bl ) for both Kakadu and our codec are: lossy or lossless com- pression as indicated, 5 levels of DWT, and codeblocks of 64 × 64. Although there are other competitive GPU imple- Algorithm 5 Kernel Routine CR({Bl }) mentations of JPEG2000 such as Comprimato [62] and 1: S ← computePosition(T , LUTL0 , L0 ) CUDA-JPEG2000 [63], it was not possible to compare them 2: if (S ∈ {L0 }) then in our test environment. Some results reported in their cor- 3: MD [o][H ] ← Bl [S] responding webpages suggest that they obtain competitive 4: else throughput, though lower to that achieved by the proposed 5: MD [o][D] ← Bl [S] codec. 6: end if For comparison purposes, the following experiments also 7: return(MD [o]) provide the throughput achieved with the HEVC implemen- tation developed by Nvidia [64], which is executed with the RTX 2080 Ti and the GTX 1080 Ti. This codec runs in before running the CR(·) kernel, so they are not specified in the GPU employing in-chip support and dedicated hardwired Algorithm 5. components. The parameters for HEVC are: rate control with Once the LUTL0 is created, each warp thread T com- constant quantization 1-51 (0) for lossy (lossless), inter-frame putes the position S of the data to be written (line 1). Then, coding with GOP=32, and high performance mode. This it checks whether the information to be copied is auxiliary configuration achieves maximum throughput in our tests. information of the codeblock (i.e., most significant bitplane), We note that HEVC is not supported in Jetson GPUs. or compressed data. This is carried out in line 2 checking The data set employed in the experiments is a 2-minute if the thread is copying the first bytes of the codeblock’s segment of the movie ‘‘Star Wars: The Last Jedi,’’ at a reso- bitstream. The corresponding bytes are either copied to the lution of 2K and 4K. The video contains 2,880 color frames header or body section of the final structure, respectively with a bit-depth resolution of 24 bits per pixel (i.e., 8 bits per denoted by MD [o][H ] and MD [o][D]. The data transfers pixel per component), resulting in 67,5 GB (16,875 GB) of are also performed in a coalesced fashion to maximize uncompressed data for the 4K (2K) resolution. The HEVC throughput. codec uses a subsampled 4:2:0 version of the video for Again, the kernels employed in the decoder are very similar compatibility issues with the 4K resolution in the GTX to those of the encoder, so they are not detailed herein. 1080 Ti. This is taken in consideration when measuring the 68482 VOLUME 8, 2020 C. de Cea-Dominguez et al.: GPU-Oriented Architecture for an End-to-End Image/Video Codec Based on JPEG2000 TABLE 4. Features of the GPUs employed. ? Both the Xavier and Tegra X2 do not have dedicated GPU memory. Memory is shared by both the CPU and GPU. FIGURE 4. Analysis of the throughput achieved by the proposed codec when coding 2K (left) and 4K (right) video using different number of execution streams, for lossy compression at maximum quality. performance achieved. In general, the size of this data set is sufficiently large to fill the resources of the GPU. Larger data sets achieve similar results as those reported below. In all results, the execution time is measured without con- sidering the I/O time spent to read/write the files from/to the disk since that would affect results significantly depend- ing on the hard drive employed. The results below evaluate only the throughput achieved since coding performance of the proposed codec is extensively analyzed in [44]. Herein, the codecs are compared when their coding options yield equivalent image quality. The first test evaluates the throughput achieved by the proposed codec with the four GPUs when using a differ- ent number of execution streams. The test evaluates both FIGURE 5. Evaluation of the average number kernels executed per unit of the encoder and decoder in lossy mode with a quantization time depending on the number of streams employed. step size that achieves maximum quality (about 50 dB). Figure 4 reports the results achieved. Again, this fig- ure depicts with horizontal lines the throughput needed to In all results, the decoder yields slightly lower throughput yield 4K, 8K, and 12K video compression in real time, assum- than the encoder because it requires more local memory. ing a frame rate of 24 frames per second. The results indicate This behavior is not common in software implementations of that both the RTX 2080 Ti and GTX 1080 Ti increase the image and video codecs since the encoder generally requires throughput as more streams are employed, yielding optimal more computations. Highly optimized implementations such performance depending on the frame resolution and GPU as the presented herein, however, may obtain different results employed. The Xavier and Tegra X2 do not benefit as much of due to the need of different data structures in the decoder. using multiple streams because they have fewer SMs, so their In the following tests, 20 and 9 streams are employed for resources are mostly filled with a single execution stream. the RTX 2080 Ti and GTX 1080 Ti, respectively, to achieve VOLUME 8, 2020 68483 C. de Cea-Dominguez et al.: GPU-Oriented Architecture for an End-to-End Image/Video Codec Based on JPEG2000 FIGURE 6. Throughput evaluation for lossy (with highest image quality) and lossless compression of 4K video, for all codecs and GPUs. Each pair of columns reports the results for the encoder (back) and decoder (front). maximum throughput. The Xavier and Tegra X2 employ slightly lower than that achieved in lossy since more data 14 and 10 streams, respectively, though their throughput is are processed, generating larger compressed files. Even so, almost the same as when using only 2. real-time 12K video can be managed by our codec for both The next test evaluates the number of kernels that are compression modes. The Xavier and Tegra X2 GPUs do not executed in parallel depending on the number of streams achieve such a high performance, but the Xavier is able to employed. This analysis complements the previous for the process 4K video in real time when employing the maximum RTX 2080 Ti. The GTX 1080 Ti, Xavier, and Tegra X2 are not performance mode. This throughput is similar to that obtained included in this analysis. Figure 5 depicts the results achieved. by Kakadu, though we recall that Kakadu employs a modern For 4K video, the maximum number of running kernels CPU and the Xavier is an embedded mobile solution. Both for is 4, which is yield when employing 10 streams. 4 parallel the Xavier and the Tegra X2, the minimum power mode sig- kernels already fill the resources of the GPU. This indicates nificantly lowers performance and the mid-tier mode achieves that no more kernels can be executed despite increasing the an intermediate performance. This is more pronounced in number of streams employed, although a slight increase in the Xavier. HEVC yields higher performance than Kakadu, throughput can be achieved as it seen in the previous figure. though it is lower than that achieved by our codec. Surpris- 2K video obtains a different behavior. Number of streams and ingly, the HEVC encoder achieves higher throughput with the running kernels are almost directly related, reaching a peak at GTX 1080 Ti than with the RTX 2080 Ti. Even though it is 20 streams and 10 parallel kernels. This is because 2K frames executed using the Nvidia SDK HEVC software (v9.0) [64] in have only a quarter of the data of 4K frames, so the GPU maximum performance mode in both, each GPU has its own requires more kernels to fill its resources. hardwired solution for this codec. More precisely, the RTX Figure 6 reports the throughput achieved by the proposed 2080 Ti includes one NVEnc Turing engine whereas the codec with the four GPUs, Kakadu, and HEVC when coding 1080 Ti includes two Pascal engines. Note also that the GTX 4K video in lossy and lossless mode. For lossy compres- 1080 Ti obtains higher throughput for the encoder than for sion, the average image quality yield for all codecs is about the decoder, whereas the RTX 2080 Ti yields more balanced 50 dB. At this level of quality, distortion is not perceptible results. by the human eye. Each codec has a pair of columns. The The previous test evaluates the performance achieved when first reports the results for the encoder whereas the second there is (almost) no quality loss. Scenarios such as video for the decoder. The results for 2K video are similar but streaming or TV broadcast may tolerate more distortion. with lower performance, so they are not included in this Reducing the image quality results in higher throughput figure. Results for the Xavier and Tegra X2 are reported since fewer data are coded. Figure 7 depicts the throughput when using three power modes, namely, maximum (0), min- achieved by Kakadu, HEVC, and the proposed codec when imum (1), and mid-tier (2) performance. The results show coding 4K video at different levels of quality, namely, from that the proposed codec yields superior performance to that 50 dB to 20 dB, which is the quality range employed in achieved by Kakadu and HEVC for both the RTX 2080 Ti most scenarios. The image quality is controlled via the quan- and GTX 1080 Ti regardless of using lossy or lossless com- tization parameter Q in our codec, and similarly in HEVC pression. In all codecs, the performance in lossless mode is and Kakadu. As seen in the figure, reducing the quality has 68484 VOLUME 8, 2020 C. de Cea-Dominguez et al.: GPU-Oriented Architecture for an End-to-End Image/Video Codec Based on JPEG2000 FIGURE 7. Throughput evaluation for lossy compression of 4K video at different quality levels. Results are for the proposed codec except when indicated. a direct impact on throughput for all codecs. The proposed codec achieves real-time encoding of 16K video for qualities below 46 dB. The decoder has a lower increase in perfor- mance as the quality decreases because the aforementioned need of more local memory. The Xavier and Tegra X2 also increase their throughput, though more gradually due to their inferior performance power. It is worth noting that, even though the RTX 2080 Ti and GTX 1080 Ti have a similar peak throughput (about 14 TFlops), the RTX 2080 Ti obtains approximately 50% more throughput when encoding. This is due to the distribution of performance power in the GPU. The RTX 2080 Ti has fewer CUDA cores in each SM, but more than twice SMs than the GTX 1080 Ti. This provides more resources per thread, especially, more local memory. Our codec greatly benefits from this architectural improvement since it employs registers extensively. The highest speedups FIGURE 8. Power consumption evaluation when encoding 4K video at reported in Figure 6 are achieved by the HEVC decoder, 50 dB. Each pair of columns reports the results for the encoder (back) and decoder (front). which increases the throughput almost 6×. Power consumption is nowadays an important aspect due to the advent of mobile devices. Figure 8 evaluates the power most. The low power consumption of our codec means that, consumption of our codec, HEVC, and Kakadu when coding in practice, it can allow batteries of mobile devices last much 4K video at 50 dB, like in Figure 6. The results are depicted longer and/or code more minutes of video for the same battery in MS processed per Watts consumed. A Nvidia tool that capacity. measures consumption in real-time is employed to obtain these results. Kakadu’s consumption is measured via the VI. CONCLUSION utility PowerTOP. The results depicted in Figure 8 suggest Faster and less power-hungry image and video codecs are cur- that the proposed codec is the most efficient in terms of power rently needed in multiple scenarios. Typically, high through- consumption. Evidently, the Xavier and Tegra X2 yield the put codecs are achieved by means of integrated hardware best results due to its architecture. Our codec employed with architectures such as ASICs or FPGAs. GPUs are also a the three power modes of these GPUs is less power-hungry widely pursued means to accelerate codecs, though these than the remaining, with the minimum mode achieving the architectures do not commonly obtain the high performance highest efficiency. The proposed codec is more efficient than of their counterparts. This is because the core algorithms of HEVC even when executed in the RTX 2080 Ti and GTX conventional image and video coding systems do not provide 1080 Ti, though moderately so. In general, CPUs consume enough fine-grain parallelism to fully exploit the SIMD archi- more power than GPUs, so Kakadu seems to consume the tecture of GPUs. This paper introduces an image/video codec VOLUME 8, 2020 68485 C. de Cea-Dominguez et al.: GPU-Oriented Architecture for an End-to-End Image/Video Codec Based on JPEG2000 based on the JPEG2000 standard. All stages of the coding [19] Y.-W. Chang, H.-C. Fang, C.-C. Chen, C.-J. Lian, and L.-G. Chen, ‘‘Word- pipeline have been devised to extract fine-grain parallelism. level parallel architecture of JPEG 2000 embedded block coding decoder,’’ IEEE Trans. Multimedia, vol. 9, no. 6, pp. 1103–1112, Oct. 2007. All stages are compliant with the standard except for the [20] M. Dyer, S. Nooshabadi, and D. Taubman, ‘‘Design and analysis of sys- core algorithm called bitplane and arithmetic coding. The tem on a chip encoder for JPEG2000,’’ IEEE Trans. Circuits Syst. Video proposed codec introduces a similar algorithm to that of Technol., vol. 19, no. 2, pp. 215–225, Feb. 2009. [21] K. Sarawadekar and S. Banerjee, ‘‘An efficient pass-parallel architecture JPEG2000 that augments its parallel capabilities. Although for embedded block coder in JPEG 2000,’’ IEEE Trans. Circuits Syst. Video the resulting codestream is not compliant with JPEG2000, Technol., vol. 21, no. 6, pp. 825–836, Jun. 2011. the coding system has the same advanced features of the stan- [22] K. Liu, E. Belyaev, and Y. Li, ‘‘A high throughput JPEG2000 entropy decoding unit architecture,’’ J. Signal Process. Syst., vol. 91, no. 8, dard. The throughput of the resulting architecture when exe- pp. 899–913, Aug. 2019. cuted in consumer-grade GPUs is at least 10× higher than that [23] Nvidia. (Dec. 2018). GPU vs CPU Theoretical GFLOP/s. [Online]. achieved with CPU implementations executed in high-end Available: https://docs.nvidia.com/cuda/cuda-c-programming-guide/ graphics/floating-point-operations-per-second.png workstations, and superior to that achieved by Nvidia’s SDK [24] M. Hopf and T. Ertl, ‘‘Hardware accelerated wavelet transformations,’’ in implementation of the HEVC video standard. Experimental Proc. IEEE Symp. Vis., May 2000, pp. 93–103. results suggest that our codec can encode (decode) real-time [25] T.-T. Wong, C.-S. Leung, P.-A. Heng, and J. Wang, ‘‘Discrete wavelet 12K (8K) video in a Nvidia RTX 2080 Ti and that it consumes transform on consumer-level graphics hardware,’’ IEEE Trans. Multime- dia, vol. 9, no. 3, pp. 668–673, Apr. 2007. very little power, especially in mobile GPUs. [26] C. Tenllado, J. Setoain, M. Prieto, L. Piñuel, and F. Tirado, ‘‘Parallel implementation of the 2D discrete wavelet transform on graphics process- REFERENCES ing units: Filter bank versus lifting,’’ IEEE Trans. Parallel Distrib. Syst., vol. 19, no. 3, pp. 299–310, Mar. 2008. [1] Digital Compression and Coding for Continuous-Tone Still Images, [27] J. Matela, ‘‘GPU-based DWT acceleration for JPEG2000,’’ in Proc. ISO/IEC Standard 10 918-1, 1992. Annu. Doctoral Workshop Math. Eng. Methods Comput. Sci., Nov. 2009, [2] N. Ahmed, T. Natarajan, and K. R. Rao, ‘‘Discrete cosine transform,’’ IEEE pp. 136–143. Trans. Comput., vol. C-23, no. 1, pp. 90–93, Jan. 1974. [28] J. Franco, G. Bernabe, J. Fernandez, and M. E. Acacio, ‘‘A parallel imple- [3] D. Huffman, ‘‘A method for the construction of minimum-redundancy mentation of the 2D wavelet transform using CUDA,’’ in Proc. IEEE Int. codes,’’ Proc. IRE, vol. 40, no. 9, pp. 1098–1101, 1952. Conf. Parallel, Distrib. Netw.-Based Process., Feb. 2009, pp. 111–118. [4] Information Technology—JPEG 2000 Image Coding System—Part 1: Core [29] W. J. van der Laan, A. C. Jalba, and J. B. T. M. Roerdink, ‘‘Accelerating Coding System, ISO/IEC Standard 15 444-1, Dec. 2000. wavelet lifting on graphics hardware using CUDA,’’ IEEE Trans. Parallel [5] I. Daubechies, Ten Lectures on Wavelets. Philadelphia, PA, USA: SIAM, Distrib. Syst., vol. 22, no. 1, pp. 132–146, Jan. 2011. 1992. [30] V. Galiano, O. Lopez, M. P. Malumbres, and H. Migallon, ‘‘Speeding- [6] F. Auli-Llinas and M. W. Marcellin, ‘‘Scanning order strategies for bitplane up the discrete wavelet transform computation with multicore and GPU- image coding,’’ IEEE Trans. Image Process., vol. 21, no. 4, pp. 1920–1933, based algorithms,’’ in Proc. Int. Conf. Comput. Math. Methods Sci. Eng., Apr. 2012. Jan. 2012, pp. 151–158. [7] High Efficiency Video Coding Standard, ITU Standard H.265, 2013. [31] V. Galiano, O. López, M. P. Malumbres, and H. Migallón, ‘‘Parallel [8] D. S. Taubman and M. W. Marcellin, JPEG2000: Image Compression strategies for 2D discrete wavelet transform in shared memory systems and Fundamentals, Standards and Practice. Norwell, MA, USA: Academic, GPUs,’’ J. Supercomput., vol. 64, no. 1, pp. 4–16, Apr. 2013. 2002. [32] P. Enfedaque, F. Auli-Llinas, and J. C. Moure, ‘‘Implementation of the [9] A. Descampe, F.-O. Devaux, G. Rouvroy, J.-D. Legat, J.-J. Quisquater, and DWT in a GPU through a register-based strategy,’’ IEEE Trans. Parallel B. Macq, ‘‘A flexible hardware JPEG 2000 decoder for digital cinema,’’ Distrib. Syst., vol. 26, no. 12, pp. 3394–3406, Dec. 2015. IEEE Trans. Circuits Syst. Video Technol., vol. 16, no. 11, pp. 1397–1410, [33] T. M. Quan and W.-K. Jeong, ‘‘A fast discrete wavelet transform using Nov. 2006. hybrid parallelism on GPUs,’’ IEEE Trans. Parallel Distrib. Syst., vol. 27, [10] T. Bruylants, A. Munteanu, and P. Schelkens, ‘‘Wavelet based volumetric no. 11, pp. 3088–3100, Nov. 2016. medical image compression,’’ Signal Process., Image Commun., vol. 31, [34] S. Datla and N. S. Gidijala, ‘‘Parallelizing motion JPEG 2000 with pp. 112–133, Feb. 2015. CUDA,’’ in Proc. IEEE Int. Conf. Comput. Elect. Eng., Dec. 2009, [11] B. Penna, T. Tillo, E. Magli, and G. Olmo, ‘‘Transform coding techniques pp. 630–634. for lossy hyperspectral data compression,’’ IEEE Trans. Geosci. Remote [35] R. Le, I. R. Bahar, and J. L. Mundy, ‘‘A novel parallel tier-1 coder for Sens., vol. 45, no. 5, pp. 1408–1421, May 2007. JPEG2000 using GPUs,’’ in Proc. IEEE 9th Symp. Appl. Specific Process. [12] Q. Huang, R. Zhou, and Z. Hong, ‘‘Low memory and low complexity VLSI (SASP), Jun. 2011, pp. 129–136. implementation of JPEG2000 codec,’’ IEEE Trans. Consum. Electron., [36] J. Matela, V. Rusnak, and P. Holub, ‘‘Efficient JPEG2000 EBCOT context vol. 50, no. 2, pp. 638–646, May 2004. modeling for massively parallel architectures,’’ in Proc. Data Compress. [13] H.-C. Fang, Y.-W. Chang, T.-C. Wang, C.-J. Lian, and L.-G. Chen, ‘‘Par- Conf., Mar. 2011, pp. 423–432. allel embedded block coding architecture for JPEG 2000,’’ IEEE Trans. [37] F. Wei, Q. Cui, and Y. Li, ‘‘Fine-granular parallel EBCOT and optimization Circuits Syst. Video Technol., vol. 15, no. 9, pp. 1086–1097, Sep. 2005. with CUDA for digital cinema image compression,’’ in Proc. IEEE Int. [14] G. Pastuszak, ‘‘A high-performance architecture for embedded block cod- Conf. Multimedia Expo, Jul. 2012, pp. 1051–1054. ing in JPEG 2000,’’ IEEE Trans. Circuits Syst. Video Technol., vol. 15, [38] M. Ciznicki, K. Kurowski, and A. Plaza, ‘‘Graphics processing unit imple- no. 9, pp. 1182–1191, Sep. 2005. mentation of JPEG2000 for hyperspectral image compression,’’ SPIE J. [15] A. K. Gupta, S. Nooshabadi, D. Taubman, and M. Dyer, ‘‘Realizing low- Appl. Remote Sens., vol. 6, pp. 1–14, Jan. 2012. cost high-throughput general-purpose block encoder for JPEG2000,’’ IEEE [39] J.-W. Lee, B. Kim, and K.-S. Yoon, ‘‘CUDA-based JPEG2000 encoding Trans. Circuits Syst. Video Technol., vol. 16, no. 7, pp. 843–858, Jul. 2006. scheme,’’ in Proc. IEEE Int. Conf. Adv. Commun. Technol., Feb. 2014, [16] Y. Li and M. Bayoumi, ‘‘A three-level parallel high-speed low-power pp. 671–674. architecture for EBCOT of JPEG 2000,’’ IEEE Trans. Circuits Syst. Video [40] X. Wu, Y. Li, K. Liu, K. Wang, and L. Wang, ‘‘Massive parallel implemen- Technol., vol. 16, no. 9, pp. 1153–1163, Sep. 2006. tation of JPEG2000 decoding algorithm with multi-GPUs,’’ Proc. SPIE, [17] Y.-W. Chang, C.-C. Cheng, C.-C. Chen, H.-C. Fang, and L.-G. Chen, vol. 9124, pp. 1–6, May 2014. ‘‘124 MSamples/s pixel-pipelined motion-JPEG 2000 codec without tile [41] F. Auli-Llinas and M. W. Marcellin, ‘‘Stationary probability model for memory,’’ IEEE Trans. Circuits Syst. Video Technol., vol. 17, no. 4, microscopic parallelism in JPEG2000,’’ IEEE Trans. Multimedia, vol. 16, pp. 398–406, Apr. 2007. no. 4, pp. 960–970, Jun. 2014. [18] K. Mei, N. Zheng, C. Huang, Y. Liu, and Q. Zeng, ‘‘VLSI design of a [42] F. Auli-Llinas, ‘‘Context-adaptive binary arithmetic coding with high-speed and area-efficient JPEG2000 encoder,’’ IEEE Trans. Circuits fixed-length codewords,’’ IEEE Trans. Multimedia, vol. 17, no. 8, Syst. Video Technol., vol. 17, no. 8, pp. 1065–1078, Aug. 2007. pp. 1385–1390, Aug. 2015. 68486 VOLUME 8, 2020 C. de Cea-Dominguez et al.: GPU-Oriented Architecture for an End-to-End Image/Video Codec Based on JPEG2000 [43] F. Auli-Llinas, P. Enfedaque, J. C. Moure, I. Blanes, and V. Sanchez, JUAN C. MOURE-LOPEZ received the B.Sc. ‘‘Strategy of microscopic parallelism for bitplane image coding,’’ in Proc. degree in computer science and the Ph.D. degree Data Compress. Conf., Apr. 2015, pp. 163–172. in computer architecture from the Universitat [44] F. Auli-Llinas, P. Enfedaque, J. C. Moure, and V. Sanchez, ‘‘Bitplane image Autònoma de Barcelona (UAB). Since 2008, coding with parallel coefficient processing,’’ IEEE Trans. Image Process., he has been an Associate Professor with the vol. 25, no. 1, pp. 209–219, Jan. 2016. Computer Architecture and Operating Systems [45] P. Enfedaque, F. Auli-Llinas, and J. C. Moure, ‘‘GPU implementation of Department, UAB, where he teaches computer bitplane coding with parallel coefficient processing for high performance architecture and parallel programming. He has par- image compression,’’ IEEE Trans. Parallel Distrib. Syst., vol. 28, no. 8, ticipated in several European and Spanish projects pp. 2272–2284, Aug. 2017. [46] D. Taubman, A. Naman, and R. Mathew, ‘‘High throughput block coding related to high performance computing. His cur- in the HTJ2K compression standard,’’ in Proc. IEEE Int. Conf. Image rent research interest focuses on the usage of massively parallel architectures Process. (ICIP), Sep. 2019, pp. 1079–1083. and the application of performance engineering techniques to open research [47] A. T. Naman and D. Taubman, ‘‘Decoding high-throughput JPEG2000 problems in bioinformatics, signal processing, and computer vision. He is (HTJ2K) on a GPU,’’ in Proc. IEEE Int. Conf. Image Process., Sep. 2019, reviewer for various magazines and symposiums and has authored numerous pp. 1084–1088. articles in journals and conferences. [48] C. D. Cea-Dominguez, P. Enfedaque, J. C. Moure, J. Bartrina-Rapesta, and F. Auli-Llina, ‘‘High throughput image codec for high-resolution satel- lite images,’’ in Proc. IEEE Int. Geosci. Remote Sens. Symp. (IGARSS), Jul. 2018, pp. 6524–6527. [49] C. de Cea-Dominguez, J. C. Moure, J. Bartrina-Rapesta, and F. Auli-Llinas, ‘‘GPU architecture for wavelet-based video coding acceleration,’’ in Parallel Computing: Technology Trends, vol. 36. Amsterdam, The Netherlands: IOS Press, Apr. 2020, pp. 83–92. [50] Nvidia. (Jan. 2018). Warp Level Primitives. [Online]. Available: https://devblogs.nvidia.com/using-cuda-warp-level-primitives/ [51] (Jun. 2019). Nvidia Tesla V100 GPU Architecture. [Online]. JOAN BARTRINA-RAPESTA received the B.Sc., Available: http://images.nvidia.com/content/volta-architecture/pdf/volta- B.E., M.S., and Ph.D. degrees in computer sci- architecture-whitepaper.pdf ence from the Universitat Autònoma de Barcelona [52] F. N. Iandola, D. Sheffield, M. J. Anderson, P. M. Phothilimthana, and (UAB), Spain, in 2002, 2004, 2006, and 2009, K. Keutzer, ‘‘Communication-minimizing 2D convolution in GPU regis- respectively, and the M.S. and Ph.D. degrees ters,’’ in Proc. IEEE Int. Conf. Image Process., Sep. 2013, pp. 2116–2120. (Hons.) from UAB. Since 2012, he has been an [53] A. Chacón, S. Marco-Sola, A. Espinosa, P. Ribeca, and J. C. Moure, Associate Professor with the Department of Infor- ‘‘Thread-cooperative, bit-parallel computation of levenshtein distance mation and Communications Engineering, UAB. on GPU,’’ in Proc. 28th ACM Int. Conf. Supercomput. (ICS), 2014, He was awarded with a Doctoral Fellowship from pp. 103–112. UAB. He participates as a reviewer for magazines [54] A. Chacon, S. M. Sola, A. Espinosa, P. Ribeca, and J. C. Moure, ‘‘FM- and symposiums, has (co)authored numerous articles in journals and confer- index on GPU: A cooperative scheme to reduce memory footprint,’’ in Proc. IEEE Int. Symp. Parallel Distrib. Process. Appl., Aug. 2014, pp. 1–9. ences, and has guided Ph.D. students. His research interests include a wide [55] W. Sweldens, ‘‘The lifting scheme: A construction of second generation range of image coding topics, computing, and transmission. wavelets,’’ SIAM J. Math. Anal., vol. 29, no. 2, pp. 511–546, Mar. 1998. [56] F. Auli-Llinas and J. Serra-Sagrista, ‘‘JPEG2000 quality scalability without quality layers,’’ IEEE Trans. Circuits Syst. Video Technol., vol. 18, no. 7, pp. 923–936, Jul. 2008. [57] Nvidia. (Dec. 2018). CUB Framework. [Online]. Available: https://nvlabs. github.io/cub/ [58] (May 2019). Jetson SDK. [Online]. Available: https://developer.nvidia. com/embedded-computing [59] D. Taubman. (Feb. 2020). Kakadu Software. [Online]. Available: http://www.kakadusoftware.com FRANCESC AULÍ-LLINÀS (Senior Member, [60] University of Stuttgart. (Jan. 2020). CuJ2K. [Online]. Available: http://cuj2k.sourceforge.net/ IEEE) received the B.E. (Hons.) and Ph.D. [61] Poznan Supercomputing and Networking Center. (Feb. 2020). GPUJ2K. (cum laude) degrees in computer science from [Online]. Available: http://apps.man.poznan.pl/trac/jpeg2k/wiki the Universitat Autònoma de Barcelona (UAB), [62] Comprimato. (Feb. 2020). Comprimato JPEG2000@GPU. [Online]. in 2002 and 2006, respectively. From 2002 to Available: http://www.comprimato.com 2015, he was consecutively funded in competitive [63] Fastvideo LLC. (Feb. 2020). CUDA-JPEG2K. [Online]. Available: fellowships, including a Ramón y Cajal Grant that https://www.fastcompression.com/products/gpu-jpeg2000.htm was awarded with the Intensification Young Inves- [64] Nvidia. (Dec. 2018). HEVC SDK. [Online]. Available: https://developer. tigator (i3) Certificate. During this time, he carried nvidia.com/nvidia-video-codec-sdk out two postdoctoral research stages with profes- sors David Taubman and Michael Marcellin. Since 2016, he has been an Associate Professor (with the Full Professor certificate) with the Department of Information and Communications Engineering, UAB. He developed and CARLOS DE CEA-DOMINGUEZ received the maintains BOI codec, a JPEG2000 implementation that is used in research B.E. and Ms.E. degrees in computer science from and professional environments. Also, he is a reviewer for magazines and Universidad de León, in 2012. He is currently symposiums, has (co)authored numerous articles in journals and confer- pursuing the Ph.D. degree with the Universitat ences, and has guided several Ph.D. students. He has participated and super- Autònoma de Barcelona. From 2012 to 2017, vised various projects funded by the Spanish government and the European he worked as a Technological Consultant. His cur- Union. His research interests lie in the area of image and video coding, rent research interests include high-performance computing, and transmission. In 2013, he received a distinguished R-Letter computing, image and video coding, and parallel given by the IEEE Communications Society for a paper co-authored with architectures. Michael Marcellin. VOLUME 8, 2020 68487