Received April 13, 2019, accepted May 14, 2019, date of publication May 31, 2019, date of current version June 13, 2019. Digital Object Identifier 10.1109/ACCESS.2019.2919970 Prediction of Energy Consumption by Checkpoint/Restart in HPC M. MORÁN 1, J. BALLADINI1 , D. REXACHS 2, AND E. LUQUE2 1 Facultadde Informática, Universidad Nacional del Comahue, Neuquén 8300, Argentina 2 Departamento de Arquitectura de Computadores y Sistemas Operativos, Universitat Autònoma de Barcelona, Campus UAB, 08193 Barcelona, Spain Corresponding author: M. Morán (marina@fi.uncoma.edu.ar) This work was supported in part by the Agencia Estatal de Investigación (AEI), Spain, in part by the Fondo Europeo de Desarrollo Regional (FEDER) UE under Contract TIN2017-84875-P, and in part by a research collaboration agreement with the Fundacion Escuelas Universitarias Gimbernat (EUG). ABSTRACT The fault tolerance method most used today in high-performance computing (HPC) is coordinated checkpointing. This, like any other fault tolerance method, adds additional energy consumption to that of the execution of the application. Currently, knowing and minimizing this energy consumption is a challenge. The objective of this paper is to propose a model to estimate the energy consumption of checkpoint and restart operations and a method for its construction. These estimates allow the evaluation of different scenarios in order to minimize energy consumption. We focus on coordinated checkpoint/restart at the system level, in single-program multiple-data (SPMD) applications, on homogeneous clusters. We study the behavior of the power dissipated by the compute node during a checkpoint/restart operation, as well as its execution time, considering different parameters of the system and the application. The experimentation carried out on two platforms shows the validity of the proposal. We also evaluate the impact on power and energy consumption of the processor’s C states, the configuration of the network file system (NFS), where the checkpoint files are stored, and the compression of the checkpoint files. This paper contributes to the objective of predicting energy consumption in the execution of applications that use checkpoint/restart. Not counting the outliers, we can estimate the energy consumed by checkpoint/restart operations with errors lower than 7.5%. INDEX TERMS Checkpointing, energy consumption, fault tolerance, high performance computing. I. INTRODUCTION application [3]. We ask ourselves, what energy saving oppor- Today’s High Performance Computing (HPC) systems use tunities does fault tolerance present? Is it possible to reduce hundreds to billions of processing units and the tendency is the energy consumption of the execution of an application by to continue increasing their computing power. However, this making its fault tolerance method more energy efficient? growth in computing power leads to an increase in energy In this article, we propose a model and a method that allows consumption. Given the limitations that exist to supply energy us to predict the energy consumption of checkpoint and to these types of computers, it is necessary to know the restart (CR) operations, considering different parameters of behavior of their energy consumption to find ways to decrease the system and the application. We focus on coordinated CR it. Specifically, there is a limit defined as a guide for exascale at the system level, in Single Program Multiple Data (SPMD) of the 20 MW [1]. applications, on homogeneous clusters. The model allows us The fault tolerance method most used today in HPC is coor- to answer questions such as, how does the compression of dinated checkpoint. With this method, the processes involved checkpoint files affect energy consumption? Compression is in the application are stopped in order to safeguard the cur- an operation that demands CPU and therefore dissipates more rent execution status in stable storage [2]. These methods power, but storing compressed files requires less transmission add additional energy consumption to the execution of the time through the network and less storage space. This model can be used to consider which parameters of the system and The associate editor coordinating the review of this manuscript and the application to use while protecting an application, in order approving it for publication was Xiaowen Chu. to decrease its total energy consumption. This work is licensed under a Creative Commons Attribution 3.0 License. For more information, see http://creativecommons.org/licenses/by/3.0/ VOLUME 7, 2019 71791 M. Morán et al.: Prediction of Energy Consumption by Checkpoint/Restart in HPC The contributions of this article are: checkpoint and on the expected failure rate in the system. • A method for constructing energy consumption predic- There are proposals on how to estimate the optimal time tion models, for coordinated CR at the system level, interval between checkpoints [5]–[7]. In this work we will in SPMD applications, on homogeneous clusters. evaluate the energy behavior of individual checkpoints and • A prediction model of energy consumption of the CR restarts, operations that are not affected by the checkpoint for a heat propagation application that follows the par- interval. allel programming paradigm SPMD. The results show that the model is able to predict the energy consump- B. DMTCP tion of CR with an error of 7.5% without considering With DMTCP (Distributed MultiThreaded Checkpointing) outliers. [8], it is possible to make checkpoints on an application in • A study of the system (hardware and software) and a transparent way. DMTCP runs at the user level, which application factors that impact the energy consumption does not require privileged access to the system to operate it. produced by the checkpoint and restart operations. In the case of parallel applications, it is possible to restart the processes in nodes which are different from those used at the The rest of the article is organized as follows. Section II time of the checkpoint, and a different number of processes introduces concepts about CR and DMTCP, which is the CR per node or a different number of nodes can be restarted. tool we use, as well as some definitions about power and DMTCP provides a coordinator, which must be permanently some ways to administer it. Section III brings together some running in any node of the cluster. From this coordinator it articles on topics related to CR and energy consumption. is possible to perform a checkpoint, abort the application, Section IV presents the theoretical model in order to know the among other functions. In our experiments, we inject the fault energy consumption. In section V we see how to obtain the by killing the processes from the coordinator. The checkpoint formulas that implement the model. We explain the results of image of each process is written to a file with a unique name, the experimentation in Section VI and finally in Section VII in a directory indicated by the user. we analyze some factors of the system and the application By using the gzip program, DMTCP is able to compress that influence the energy consumption of CR operations. the process state to require less disk storage space and reduce Section VIII has the conclusions. the amount of data transmitted over the network (between the compute node and the storage node). It is possible to II. BACKGROUND indicate if we want to use compression or not by using an In this section, we introduce some concepts about coordi- environment variable, or with an option of the dmtcp_launch nated CR and DMTCP, which is the CR tool used, and we command. The dmtcp_launch command is the command used give some definitions related to the power and the ways to to execute an application. The first call to dmtcp_launch will administer it. launch the coordinator, if it did not already exist. For example, to launch an MPI application with 8 processes and without A. CHECKPOINT/RESTART using compression, we can use: Currently, the fault tolerance method most widely used in $dmtcp_launch -no-gzip mpirun -np 8 ./pr large computer systems is coordinated CR. With this method, the consistent global state is guaranteed by synchronizing all To perform a checkpoint manually from the command line the processes at the time of the checkpoint. Faced with the we use: failure of a process, all processes must restart from the last $dmtcp_command c checkpoint. The wide use of this method is due to its simple implementation, since the synchronization points required to To restart the application after a failure, the script generated store the state of the processes are present in most parallel by DMTPC is used as follows: MPI applications. $sh dmtcp_restart_script.sh The recovery of failures through checkpoints consists mainly of restarting the application replacing the component As we will see later, we are interested in knowing the that has failed, but to avoid a complete re-execution of the average power dissipated and the time of each CR operation. application, the state of execution is periodically saved in These values depend on several factors, including the system stable, local or remote, storage. When a fail-stop failure is architecture, the file system, the application, and the fault detected, the application is restarted from the last stored tolerance system used. In particular, from the DMTCP’s point execution state [2]. In the case of parallel applications, we can of view, the variables that the user can modify and that affect define a consistent global checkpoint as the set of local power and time are: checkpoints, one for each process, which form a consistent • Application problem size: The totality of the memory global state. Any consistent global checkpoint can be used to assigned to the user process must be safeguarded. restart the execution of an application after a failure [4]. • Whether compression is used or not: A compressed file Checkpoints have to be taken periodically. The time inter- takes less time to store, and compression dissipates more val between checkpoints depends on the duration of the power. 71792 VOLUME 7, 2019 M. Morán et al.: Prediction of Energy Consumption by Checkpoint/Restart in HPC is launched. Finally, at the end of the restart, the execution of the application continues, at time ‘f’. D. POWER ADMINISTRATION: P AND C STATES The ACPI specification (Advanced Configuration and Power Interface) provides an open standard that allows the operating system to manage the power of the devices and the computing system.1 It allows us to manage the energy behavior of the processor, the component that consumes the most energy in a computer system. ACPI defines Processor Power States (Cx states), where C0 is the execution state, and C1. . . Cx are inactive states. A processor that is in the C0 state will also be in a Perfor- FIGURE 1. Real power dissipated during checkpoint, restart, application, mance State (Px states). The P0 state means an execution at and idle time. the maximum capacity of performance and power demand. • Whether incremental checkpoint is used or not: As the number of P state increases, its performance and An incremental checkpoint has a smaller size than a full demanded power is reduced. The processors implement the P one. states using the technique of Dynamic Frequency and Voltage Scaling (DVFS) [9]. Reducing the voltage supply reduces the C. POWER DEFINITIONS energy consumption. However, the delay of the logic gates increases, so it is necessary to reduce the clock frequency of In this subsection we define some power related terms used the CPU in order for the circuit to work correctly. In certain throughout the article. multicore processors, each core is allowed to be in a different • Instantaneous power: This is the power absorbed by P state. an element at any instant of time, and is given by When there are no instructions to execute, the processor the product of instantaneous voltage (V ) and instan- can be set in a state C greater than 0 to save energy. There taneous current (I ), which pass through that element, are different levels of C states, where each of the levels could as shown in 1. In this work, this power is obtained from turn off certain clocks, reduce certain voltages supplied to idle the voltage and current measurements delivered by the components, turn off the cache memory, etc. The higher the oscilloscope. number of the C state, the lower the power demanded, but a P[Watts] = I [Amperes] × [Volts] (1) high latency is required to return to state C0 (execution state). In certain multicore processors, each core is allowed to be in • Active or real power: This is the average of the instan- a different C state.2 taneous power over an alternating current (AC) com- plete cycle. In AC, a sine wave represents the value of III. RELATED WORK voltage and current through time. In countries where Some studies evaluate the energy behavior of the coordi- the frequency is 50Hz, the change of direction occurs nated and uncoordinated CR with message logs. In [10] they 50 times per second, giving a duration of the complete compare the power dissipated by saving the message logs in cycle of 20 milliseconds. RAM and in HDD. They measure the consumption of atomic • Average power: This is the average of the instantaneous operations found in the coordinated and uncoordinated pro- power in a given period of time. In our case, we will tocols. In [11] they also evaluate the parallel recovery and average the real power for the duration of the checkpoint propose an analytic model to predict the energy behavior or restart. From now on, when we mention power or at exascale. They perform experiments on a cluster to mea- dissipated power, we mean average dissipated power. sure the consumption and then compare with the energy • Base power: The power dissipated by the node when the estimated with the model. They show how parallel recovery processor is idle. can reduce the execution time and energy consumed in a • Dynamic power: This is the power dissipated without faulty scenario. Parallel recovery is based on task migra- considering the base power. tion, so they use Charm++ instead of pure MPI. In this Fig. 1 shows the graph of the real power, obtained with work we measure the complete checkpoint operations on the measurements delivered by the oscilloscope, when an MPI applications. application is executed and a fault is injected (to see how the There are some other studies that do not take power fault is injected, refer to subsection II-B). The execution of measures. In [12] an analytic model is used to compare the the application starts at time ‘a’. At time ‘b’ the checkpoint begins, and it finishes at time ‘c’. The fault is introduced at 1 http://www.acpi.info time ‘d’, and a period of inactivity begins there (we can see 2 https://software.intel.com/en-us/articles/power-management-states-p- that the base power is around 50 W). At time ‘e’, the restart states-c-states-and-package-c-states VOLUME 7, 2019 71793 M. Morán et al.: Prediction of Energy Consumption by Checkpoint/Restart in HPC execution time and the consumed energy of the replication and the coordinated CR. References [13] and [14] use ana- lytic models to estimate the optimal interval of a multilevel checkpoint in terms of energy consumption. They do not mea- sure dissipated power but use values from other publications. An optimization problem is presented in [15], finding the clock frequency that minimizes the power consumption of an application that uses shadow processes as a fault tolerance method. Each main process has its corresponding shadow process, which executes the same program as the main one, but at a slower clock frequency. When a main process fails, the shadow replaces it, and the frequency at which it should continue executing the new main process is calculated in FIGURE 2. Power dissipated during checkpoint and restart for different order to meet a stipulated maximum time limit. It is designed CPU frequencies. primarily for cloud, and it performs all the analysis in an analytic manner. Our work focuses on the prediction of the energy con- A framework to estimate the energy consumption of coor- sumption of the system level coordinated CR, for an SPMD dinated, uncoordinated and hierarchical checkpoints is con- application. The dissipated power values of the checkpoint sidered in [16]. They measure the power and the execution and restart operations are measurements obtained with an time of the high level operations involved in the checkpoint, external meter that measures the complete node, including the varying the number of cores involved. They do not use differ- source. We evaluate the impact that the CPU clock frequency, ent processor frequencies, nor do they indicate whether the the problem sizes, the processor C states, the compression of checkpoint is compressed or not. the checkpoint files and the NFS configuration all have on the References [17], [18] and [19] all present a framework for energy consumption of the CR. CR energy saving. In [17], many small I/O operations are replaced by a few large ones, executing in a single core, IV. ENERGY MODEL to make the checkpoint and restart more energy efficient. Energy can be calculated as the product between power and They use RAPL to measure and limit energy consumption. time, as in 2. Reference [18] proposes having a core to execute a replica of E[Jules] = P[w] × T [s] (2) all the processes of the node in order to avoid re-execution from the last checkpoint and analytically compare the energy We need to know the average power dissipated and the consumption of this proposal with the traditional check- time of each operation, checkpoint or restart. The power, point. Another paper that focuses on I/O is [19]. Here, they P[w], is obtained by averaging the instantaneous power first characterize different storage media by executing a (section II-C) corresponding to the execution period of check- microbenchmark that reads and writes large files, and then point or restart. The time, T [s], is obtained from DMTCP they design a runtime that allows modifying the clock fre- (section II-B). quency and the number of processes that perform the I/O Now, what variables affect power? And what variables to optimize energy consumption according to the type of affect time? Next, we analyze these variables to determine storage used. In our case, we evaluate the energy behavior which ones are significant for our model. of a specific checkpoint library. Another study analyzing the impact of dynamic scaling A. CLOCK FREQUENCY of frequency and voltage on the energy consumption of The CPU clock frequency affects the voltage supplied, and checkpoint operations is [20]. They measure the power at therefore, affects the dynamic power [22]. If the checkpoint the component level while writing local and remote check- files are compressed, compression is a CPU intense opera- point files. They also compare two remote storage media: tion. In Fig. 2 we observe that the higher the clock frequency, NFS using the kernel network stack and NFS using the IB the higher the power dissipated. RDMA interface. They do not include the restart in the With regards to time, in Fig. 3, we observe how the clock measurements. frequency especially affects the checkpoint time, whereas the In [21] they evaluate the energy consumption of an applica- restart time is much less affected. tion that uses compressed checkpoints. They show that using compression requires more power but saves time, so that B. PROBLEM SIZE the complete execution of the application with all its check- Problem size refers to the size of the memory assigned to points can benefit in terms of energy. They do not modify the user process. As we can see in Fig. 4, problem size has the frequency of the processor. Like us, they measure the little impact on the dissipated power during a CR operation. power dissipated by the entire checkpoint operation (and not On the other hand, as the system level checkpoint has to save subtasks or suboperations). the total memory assigned to the user process (section II-A), 71794 VOLUME 7, 2019 M. Morán et al.: Prediction of Energy Consumption by Checkpoint/Restart in HPC In this way, we obtain the following general equations of the energy consumed per node, during checkpoint, EC , and during restart, ER : EC = PC (f ) × TC (f , ps) (3) ER = PR (f ) × TR (f , ps) (4) where f is the clock frequency, ps is the problem size per node, PC and PR indicate the average dissipated power during a checkpoint and restart operation respectively, and TC and TR indicate the checkpoint and restart time respectively. FIGURE 3. Time consumed by checkpoint and restart for different CPU frequencies. V. ENERGY CONSUMPTION ESTIMATION In this section we show how to build the model to estimate power and time. This construction consists of two steps: collecting data through the characterization of the system, and regression analysis. When characterizing the system, we want to know how it behaves with respect to power and time while performing checkpoint and restart. Some considera- tions on this characterization are presented in the following subsection. The regression analysis allows us to obtain the functions to build the model and to make the estimations. Subsection V-B shows the regression analysis performed on the data obtained from the experimentation on Platform 1. A. SYSTEM CHARACTERIZATION We want to be able to characterize the system with the lowest FIGURE 4. Power dissipated during checkpoint and restart for different possible cost, that is, with a small set of tests. The following problem sizes. questions arise; what to measure and how to measure it? Mea- suring the power of the cluster nodes requires the connection of external instruments, and accesses to the cluster that can be expensive. The number of repetitions of the experiment should be considered according to the variability of the mea- surement instruments. In the following subsections, we describe some consid- erations about the duration of the preheating and the fre- quencies, problem sizes and application to be used in the characterization. 1) PREHEATING Before starting to measure the dissipated power, the processor FIGURE 5. Time consumed by checkpoint and restart for different must be preheated, that is, the application must be left running problem sizes. until it reaches the maximum levels of power dissipated. This period can be defined empirically, observing when the the time of a CR is affected by the problem size, as shown dissipated power curve reaches the maximum values for the in Fig. 5. application. C. NUMBER OF CORES 2) CLOCK FREQUENCIES SELECTION The number of cores used to execute the CR affects both The CPU frequencies for the experiments have to be defined. power and time. However, as we use the computation node Because the range of clock frequencies is discrete and limited, as a minimum unit, we did not include this variable in this if within the selected frequencies the minimum and maximum study. frequencies are included, the power function obtained, PCR , In summary, we will evaluate the behavior of dissipated will show the behavior of the remaining frequencies. In the power and time varying on the one hand, the frequency of case that there is no curve that adequately represents the the processor, and on the other hand, the size of the problem. power, a split function can be defined. In any case, this VOLUME 7, 2019 71795 M. Morán et al.: Prediction of Energy Consumption by Checkpoint/Restart in HPC would result in a large number of experiments, and con- sequently, longer access to the cluster with the measuring instruments connected. If we have 13 different frequencies, and we do three repetitions of each experiment, we need almost 40 experiments. In this paper we will show how it is possible to obtain good predictions of the dissipated power by measuring only four frequencies, reducing the number of experiments by 70%. 3) PROBLEM SIZES SELECTION The problem sizes for the experiments have to be defined. In applications that do not vary the amount of memory used throughout their execution, it will be enough selecting the size of the problem before each experiment. The selected problem sizes must not exceed the capacity of the main memory, to avoid the use of swap. If there are no user requirements for too-small problem sizes, it is preferable to avoid them, as they show greater variability. 4) APPLICATION SELECTION System characterization is done for different problem sizes, therefore the selected application must allow to change this value. Problem size refers to the size of the program data that is stored in main memory. Checkpoint and restart operations save the program state, which is mainly program data, and are not affected by the application executed by the processor. Due to this, the consumption of the application does not intervene in the construction of the model and it is sufficient to char- acterize the system using a single application. Applications that allow to change the problem size in a simple and flexible manner are preferred. FIGURE 6. Power measurements and regression. (a) Checkpoint. (b) Restart. B. REGRESSION ANALYSIS: OBTAINING FUNCTIONS AND COEFFICIENTS Fig 6 also shows the regression obtained. As we can see, The power and time measurements obtained must be analyzed the measurements are mostly within the range of +/− two to find the function that best approximates them, and then standard deviations (SD) from the curve. The exceptions are the coefficients of this function are obtained by the least given, for the case of the checkpoint, at the maximum fre- squares method. In the following subsections, this analysis quency, and in the case of the restart, at the lowest frequency. is shown (see section VI-A for the experimental platform The average of the absolute errors can be considered low used). (3.2W for the checkpoint and 2W for the restart), as well as the standard deviations (4.6W for the checkpoint and 2.4W 1) POWER for the restart). It is worth mentioning that the standard devi- Fig. 6 shows the dissipated power during the CR for ation of the checkpoint is almost twice the standard deviation the four clock frequencies and the four per node prob- of the restart, which indicates a greater variability. lem sizes selected. When observing this cloud of points, Fig. 7 shows the corresponding box graph. A few out- we see that a linear or quadratic function could represent liers can be observed. The outliers are values that are more it. Applying the least squares method with both functions, than one and a half times away from the inter-quartile we obtained smaller errors with the quadratic function, both range. Another interesting observation is that checkpoint and for checkpoint and restart. The formulas obtained are the restart higher boxes, that is, the greater dispersion of data, following: are given for the case of the maximum clock frequency (2.667 GHz). PC (f ) = 19.7 × f 2 − 37.7 × f + 99 (5) 2 2) TIME PR (f ) = 9.55 × f − 18.62 × f + 88.45 (6) Fig. 8 shows the execution times obtained for the four clock where f is clock frequency and its unit is in GHz. frequencies and the four per node problem sizes. When 71796 VOLUME 7, 2019 M. Morán et al.: Prediction of Energy Consumption by Checkpoint/Restart in HPC FIGURE 8. Time measurements and regression. (a) Checkpoint. FIGURE 7. Power measurements box plots. (a) Checkpoint. (b) Restart. (b) Restart. observing this cloud of points, we see that a linear or quadratic 2.25 GiB), errors in the energy estimation are calculated, and function could represent it. We applied the least squares the dissipated power on the two experimental platforms is method on both functions and we obtained smaller errors with compared. the quadratic function, both for checkpoint and restart. The formulas obtained are the following: A. EXPERIMENTAL PLATFORMS 2 2 TC (ps, f ) = 1.42 × ps + 6.07 × f + 23.74 × ps The experiments were carried out on two platforms. Plat- form 1, on which most of the analysis of this work is carried − 23.87 × f − 7.67 × ps × f + 26.97 (7) out, and Platform 2, which is used to contrast the results 2 TR (ps, f ) = 2.11 × f + 7.45 × ps − 9.31 × f − 0.46 obtained with a different hardware. Platform 1 is a cluster of × ps × f + 10.75 (8) computers, with a 1 Gbps Ethernet network. Each node, both computing and storage, has 4 GiB of main memory, a SATA where f is clock frequency in GHz and ps is the problem hard disk of 500 GB and 7200 rpm, and an Intel Core i5- size per node in GiB. Fig. 8 also shows the regression, 750 processor, with a frequency range of 1.2 GHz to 2.66 GHz where 94% and 96% of the measurements fall within the (with the Intel Turbo Boost3 disabled), four cores (without band defined by +/− two standard deviations for the case multithreading), 8 MiB of cache and 95 W TDP. Platform of the checkpoint and restart respectively. The correlation 2 is a compute node connected to a storage node with a coefficient obtained is high (0.99 in both cases, checkpoint 1 Gbps Ethernet network. The compute node has an Intel and restart), and the standard deviations give very accept- Xeon E5-2630 processor, a frequency range of 1.2 GHz to able values (1 s for the checkpoint and 0.42 s for the 2.801 GHz (with the Intel Turbo Boost mechanism disabled), restart). six cores (with multithreading disabled), 16 GiB of main memory, 15 MiB of cache and TDP of 95 W. It uses a Debian VI. RESULTS ANALYSIS 9 ‘‘Stretch’’ operating system. The storage computer has In this section, the experimental platforms and experimental an Intel Core 2 Quad Q6600 processor, four cores (without design are detailed, the power and time models are vali- dated for new clock frequencies (1.466 GHz, 1.999 GHz and 3 https://www.intel.com/content/www/us/en/architecture-and- 2.533 GHz) and new problem sizes (1 GiB, 1.5 GiB and technology/turbo-boost/turbo-boost-technology.html VOLUME 7, 2019 71797 M. Morán et al.: Prediction of Energy Consumption by Checkpoint/Restart in HPC multithreading), 8 GiB of main memory and 4 MiB of cache memory. The nodes of Platform 1 and the computing node of Platform 2 use the GNU/Linux operating system Debian 8.2 Jessie (kernel version 3.16 of 64 bits), OpenMPI ver- sion 1.10.1 as an MPI message passing library, and the tool checkpoint DMTCP version 2.4.2, configured to compress the checkpoint files. The network file system used to make the remote writing of the checkpoint files is NFS v4 (Network File System). For power measurements we use the PicoScope 2203 oscil- loscope (whose accuracy is 3%), the TA041 active differential probe, and the PP264 60 A AC/DC current clamp, all Pico Technology products. The electrical signals captured by the two-channel oscilloscope are transmitted in real time to a computer through a USB connection. The voltage is mea- sured using the TA041 probe that is connected to an input channel of the oscilloscope. The current of the phase con- ductor, which provides energy to the complete node (includ- ing the power source) is measured using the current clamp PP264, which is connected to the other input channel of the oscilloscope. The selected application4 for system characterization is a SPMD heat transfer application written in MPI that uses the float data type. This application describes, by means of an equation, the change of temperature in time, on a plane, given an initial temperature distribution and certain edge FIGURE 9. Checkpoint and restart power regression validation. conditions. (a) Checkpoint. (b) Restart. 4.2.6 Limits of the formalization. El segundo párrafo sep- ara lo que es estrictamente ‘‘tamaño del problema’’ y ‘‘tra- bajo/dificultad’’. Entonces, en el artículo se podría aclarar by DMTCP. The experiment is repeated three times for each que aquí no hacemos referencia a la ‘‘dificultad’’ de la tarea frequency and problem size due to the low variability of the sino al tamaño de los datos que se almacenan en memoria measurement instruments used. principal. C. MODEL VALIDATION B. EXPERIMENTAL DESIGN 1) POWER VALIDATION Each compute node writes to a dedicated storage node In order to validate our power model, we measured the through an NFS configured in asynchronous mode. The dissipated power for new clock frequencies and calculate sampling rate used for both channels of the oscilloscope the errors obtained. Fig. 9 shows the graph with the regres- was set at 1000 Hz. The power measurements correspond sion and the new measurements. We can observe how most to the power dissipated by the complete node including of the new values fall within the range between +/− one the source. The tests were performed with the proces- standard deviation. The average of the errors’ absolute val- sor C states option active. For the measurements of the ues is low (3.7 W for the checkpoint and 4 W for the checkpoint and restart time, we use the option provided restart), and the standard deviations increase slightly when by DMTCP. To change the frequency of the processor, compared with the previous experiments (5.4 W for the the GNU/Linux userspace governor is used, which permits checkpoint and 5.1 W for the restart), but they are still modifying the file sysfs scaling_setspeed available for each acceptable. core. The same frequency is used in all cores at the same time. Each experiment consists of launching the application with POWER ON PLATFORM 2 one process per core, letting it run during a preheating period In order to observe the behavior of the power dissipated in (20 seconds), performing a checkpoint manually, aborting the a different computer, the same experiments were performed application from the DMTCP coordinator and re-starting the on Platform 2. Fig. 10 shows the initial observations (marked application from the command line with the script generated with circles), the regression function that best fits those obser- vations, and validation measurements (marked with squares). 4 https://computing.llnl.gov/tutorials/parallel_comp/#ExamplesHeat As we can see, most measurements fall within the band 71798 VOLUME 7, 2019 M. Morán et al.: Prediction of Energy Consumption by Checkpoint/Restart in HPC FIGURE 10. Platform 2 power linear regression and validation. TABLE 1. Power on Platform 2: Correlation coefficient, standard deviations and average of absolute errors. defined by +/− two standard deviations. Table 1 shows the indicators that validate the obtained regressions. We will mention two differences with respect to the results obtained with Platform 1. On the one hand, in this case it is a lineal function that best fits the point cloud of the power, whereas on Platform 1 it was a curve. On the other FIGURE 11. Real power dissipation during checkpoint and restart. hand, on this platform we observed that there is a rela- (a) Platform 1 at 2.667 GHz. (b) Platform 2 at 2.5 GHz. tionship between power and problem size. For example, in the checkpoint case, we see large problem sizes to be TABLE 2. Energy consumption prediction: Percentage of measurements mostly above the regression line, and small sizes, below. that fall within +/- one and two standard deviation. In any case, the estimation obtained is good enough and we don’t consider it justified to make the model more complex. Fig. 11 shows the real power of Platform 2. It presents a high and a low phase during the checkpoint, not present in Platform 1. The high phase indicates that the CPU is 3) ENERGY VALIDATION compressing, and the low phase that it is transferring and has We will now calculate the errors obtained in the predic- finished compressing. In graphics not shown here, we can tion of the CR energy consumption. Fig. 12 shows the see how the high phase becomes shorter as the frequency prediction function of the energy and the values observed increases (since it compresses faster). in the experiment. More than 90% of the measurements fall within the range defined by +/− two standard devi- 2) TIME VALIDATION ations (for both checkpoint and restart), as summarized In order to validate the model for the CR time, measure- in Table 2. ments were made for new frequencies and problem sizes, and Table 3 shows the average error incurred for each size the errors obtained were calculated. The errors in absolute and frequency, in the prediction of the energy consumption. value have an average of 0.7 seconds for the checkpoint and In the case of the checkpoint, the highest error percentages are 0.4 seconds for the restart. The standard deviations are kept around 6%, except for one case, corresponding to the smallest low, these being 1.03 seconds for the checkpoint and 0.6 sec- size and the maximum frequency (24.15%). In the case of the onds for the restart. With these new measurements, we can restart, the highest error percentages are around 7.5%, except verify that the proposed model can predict CR time very for two cases, corresponding to the smallest size (10.54% and well. 12.16%). In this way, without taking outliers, we have been VOLUME 7, 2019 71799 M. Morán et al.: Prediction of Energy Consumption by Checkpoint/Restart in HPC FIGURE 12. Energy measurements and prediction. (a) Checkpoint. FIGURE 13. Power dissipated with C states enabled and disabled. (b) Restart. (a) Checkpoint.(b) Restart. TABLE 3. Average errors (in %) by frequency and problem size. TABLE 4. Predicted and measured energy (1.25 GiB problem size). case, the frequency with the lowest energy consumption is 2.266 GHz. The table also shows the errors incurred. VII. FACTORS THAT AFFECT THE ENERGY CONSUMPTION able to estimate the energy consumed by the CR with an error AND/OR THE PREDICTION QUALITY of less than 7.5%. In this section we analyze some system and application fac- tors that influence the energy consumption of CR operations. USE OF THE PREDICTION MODEL FOR THE CHECKPOINT ENERGY CONSUMPTION A. C STATES Table 4 shows the prediction of the checkpoint energy con- During the writing or reading of a checkpoint file it is possible sumption for a fixed problem size and ten clock frequen- that the processor becomes idle and therefore transitions cies. This allows us to know what clock frequency to use occur between the C states (section II-D) that affect the if we decide to prioritize the energy consumption. In this power dissipation of the processor. To study its behavior, 71800 VOLUME 7, 2019 M. Morán et al.: Prediction of Energy Consumption by Checkpoint/Restart in HPC FIGURE 14. Power dissipated, time and energy consumed during checkpoint with sync and async NFS mode. CR operations were performed with C states enabled and dis- It was also observed that the checkpoint times (for the same abled, for all processor frequencies. In Fig. 13 it is observed problem size and clock frequency) show greater variability in that the power measurements with the C states enabled show the synchronous mode, differences of up to 40% (in this case greater variability, especially in the restart. This variability we carry out more experiments), when compared to the asyn- affects the prediction. In any case, the best option is to keep chronous mode, differences less than 1%. The low variability the C states enabled, since they reduce the energy consump- of the checkpoint time with the asynchronous mode benefits tion by up to 13% for the checkpoint, and up to 20% for its prediction. the restart. The execution times showed no variation when Due to its lower dissipated power, and its lower time enabling or disabling the C states. variability, it is preferable to store the checkpoints in an NFS mounted with the async option. B. NFS CONFIGURATION The NFS allows folders to be mounted synchronously (sync C. COMPRESSION OF CHECKPOINT FILES option) or asynchronously (async option). If an NFS folder The compression or not of the checkpoint files impacts the is mounted with the sync option, the writes at that mount time and the power demanded, therefore it is another factor point will cause the data to be completely downloaded to the that affects the energy consumption. NFS server and written to persistent storage before returning Fig. 15 compares the power dissipated, the time and the control to the client.5 Thus, the time of a write operation is energy consumed by the checkpoint and restart, with and affected by varying this configuration. without compression of the checkpoint files, for three dif- Fig. 14 shows the comparison of the power dissipated, ferent clock frequencies (minimum, medium and maximum the execution time and the energy consumed by a checkpoint values available in the processor). The experiments were stored on a network file system mounted with the option sync performed on a single compute node, writing to a single and async, for three different clock frequencies (minimum, storage node. We can make the following observations when medium and maximum available in the processor). For all using compression compared with no compression: three frequencies, the dissipated power is greater and the • With regard to power: The dissipated power and its execution time is shorter when the asynchronous configura- variability are greater, both for checkpoint and restart. tion is used. This is because the idle time of the processor • With regard to time: The time variability is greater, decreases when using this configuration (and therefore the both for checkpoint and restart. The restart is always average power increases). faster. The checkpoint is faster only for the maximum For the small and middle frequency, the differences in time frequency and slower for the minimum frequency. and power are small, resulting in a similar energy consump- • With regard to energy: The checkpoint consumes more tion. However, for the maximum frequency, this difference energy (up to 55%) and the restart consumes less energy gets higher, reaching an energy saving of up to 25% when (up to 20%). using the asynchronous mode. Taking into account that, in general, checkpoint is per- formed many more times than restart, it is advisable not to use 5 https://linux.die.net/man/5/nfs compression to reduce energy consumption in this platform. VOLUME 7, 2019 71801 M. Morán et al.: Prediction of Energy Consumption by Checkpoint/Restart in HPC In addition, the impact of different system and application factors on energy consumption was evaluated in an experi- mental platform. In particular, the impact of the processor’s C states, the synchronous and asynchronous configuration of the NFS, and the compression or not of the checkpoint files were evaluated. We saw how it is better, from an energy point of view, to use an asynchronous configuration of the NFS to enable the C states of the processor and so as not to compress the checkpoint files. These results show how it is possible to influence the energy consumption of CR. This work contributes to the objective of predicting energy consumption in the execution of applications that use CR. Not counting the outliers, we can estimate the energy consumed by the CR with errors lower than 7.5%. Among future work it is expected to evaluate the energy behavior of other types of applications, of uncoordinated CR, as well as to evaluate the energy consumed by the storage node. REFERENCES [1] K. Bergman, S. Borkar, D. Campbell, W. Carlson, W. Dally, M. Denneau, P. Franzon, W. Harrod, K. Hill, J. Hiller, and S. Karp, ‘‘Exascale computing study: Technology challenges in achieving exascale systems,’’ Defense Adv. Res. Projects Agency, Arlington, VA, USA, Tech. Rep. 15, 2008. [2] E. N. Elnozahy, L. Alvisi, Y.-M. Wang, and D. B. Johnson, ‘‘A survey of rollback-recovery protocols in message-passing systems,’’ ACM Comput. Surv., vol. 34, no. 3, pp. 375–408, 2002. [3] J. Shalf, S. Dosanjh, and J. Morrison, ‘‘Exascale computing technology challenges,’’ in Proc. Int. Conf. High Perform. Comput. Comput. Sci. Springer, 2010, pp. 1–25. [4] S. Sankaran, J. M. Squyres, B. Barrett, V. Sahay, A. Lumsdaine, J. Duell, P. Hargrove, and E. Roman, ‘‘The LAM/MPI checkpoint/restart frame- work: System-initiated checkpointing,’’ Int. J. High Perform. Comput. Appl., vol. 19, no. 4, pp. 479–493, 2005. [5] J. W. Young, ‘‘A first order approximation to the optimum checkpoint interval,’’ Commun. ACM, vol. 17, no. 9, pp. 530–531, 1974. [6] J. T. Daly, ‘‘A higher order estimate of the optimum checkpoint interval for restart dumps,’’ Future Gener. Comput. Syst., vol. 22, no. 3, pp. 303–312, 2006. [7] N. El-Sayed and B. Schroeder, ‘‘To checkpoint or not to checkpoint: Understanding energy-performance-I/O tradeoffs in HPC checkpoint- ing,’’ in Proc. IEEE Int. Conf. Cluster Comput. (CLUSTER), Sep. 2014, pp. 93–102. [8] J. Ansel, K. Arya, and G. Cooperman, ‘‘DMTCP: Transparent checkpoint- ing for cluster computations and the desktop,’’ in Proc. IEEE Int. Symp. Parallel Distrib. Process., May 2009, pp. 1–12. [9] E. Le Sueur and G. Heiser, ‘‘Dynamic voltage and frequency scaling: FIGURE 15. Comparison of compression/no compression of checkpoint The laws of diminishing returns,’’ in Proc. Int. Conf. Power Aware Comput. files. (a) Power dissipated. (b) Time consumed. (c) Energy consumed. Syst., 2010, pp. 1–8. [10] M. el Mehdi Diouri, O. Glück, L. Lefevre, and F. Cappello, ‘‘Energy considerations in checkpointing and fault tolerance protocols,’’ in Proc. VIII. CONCLUSIONS IEEE/IFIP Int. Conf. Dependable Syst. Netw. Workshops (DSN), Jun. 2012, This work shows how it is possible to estimate the energy pp. 1–6. consumption of system level coordinated CR operations, [11] E. Meneses, O. Sarood, and L. V. Kalé, ‘‘Energy profile of rollback- on an SPMD application executed in a homogeneous cluster. recovery strategies in high performance computing,’’ Parallel Comput., vol. 40, no. 9, pp. 536–547, 2014. We proposed a method for constructing a model to predict [12] B. Mills, T. Znati, R. Melhem, K. B. Ferreira, and R. E. Grant, ‘‘Energy the dissipated power and time of CR operations. The method, consumption of resilience mechanisms in large scale systems,’’ in Proc. based on regression analysis, was applied and validated in 22nd Euromicro Int. Conf. Parallel, Distrib., Netw.-Based Process., Feb. 2014, pp. 528–535. an experimental platform. Besides, a regression analysis and [13] M. A. Amrizal and H. Takizawa, ‘‘Optimizing energy consumption on HPC validation was performed for the dissipated power of a dif- systems with a multi-level checkpointing mechanism,’’ in Proc. Int. Conf. ferent machine. With this model, it is possible to know the Netw., Archit., Storage (NAS), Aug. 2017, pp. 1–9. energy consumption for new CPU frequencies and problem [14] D. Dauwe, R. Jhaveri, S. Pasricha, A. A. Maciejewski, and H. J. Siegel, ‘‘Optimizing checkpoint intervals for reduced energy use in exascale sys- sizes without the need to execute the checkpoint and restart tems,’’ in Proc. 8th Int. Green Sustain. Comput. Conf. (IGSC), Oct. 2017, of the application and to take measurements. pp. 1–8. 71802 VOLUME 7, 2019 M. Morán et al.: Prediction of Energy Consumption by Checkpoint/Restart in HPC [15] B. Mills, T. Znati, and R. Melhem, ‘‘Shadow computing: An energy- J. BALLADINI received the Ph.D. degree in com- aware fault tolerant computing model,’’ in Proc. Int. Conf. Comput., Netw. puter science from the University Autònoma of Commun. (ICNC), Feb. 2014, pp. 73–77. Barcelona, Spain, in 2008. He is currently a Pro- [16] M. el Mehdi Diouri, O. Glück, L. Lefevre, and F. Cappello, ‘‘ECOFIT: fessor and the Director of the Computer Engineer- A framework to estimate energy consumption of fault tolerance protocols ing Department, National University of Comahue, for HPC applications,’’ in Proc. 13th IEEE/ACM Int. Symp. Cluster, Cloud, Argentina. He is also the Secretary of Science Grid Comput., May 2013, pp. 522–529. and Technology of the Network of Argentinian [17] R. R. Chandrasekar, A. Venkatesh, K. Hamidouche, and D. K. Panda, National Universities with degrees in Computer ‘‘Power-check: An energy-efficient checkpointing framework for HPC Science (RedUNCI). His major research inter- clusters,’’ in Proc. 15th IEEE/ACM Int. Symp. Cluster, Cloud Grid Com- put., May 2015, pp. 261–270. ests include high-performance computing (HPC), [18] X. Cui, T. Znati, and R. Melhem, ‘‘Adaptive and power-aware resilience energy consumption of HPC systems, and real-time big data processing in for extreme-scale computing,’’ in Proc. Int. IEEE Conf. Ubiquitous Intell. critical care units. Comput., Adv. Trusted Comput., Scalable Comput. Commun., Cloud Big Data Comput., Internet People, Smart World Congr. (UIC-ATC-ScalCom- D. REXACHS received the Ph.D. degree in com- CBDCom-IoP-SmartWorld), Jul. 2016, pp. 671–679. doi: 10.1109/UIC- puter science from the University Autònoma of ATC-ScalCom-CBDCom-IoP-SmartWorld.2016.0111. Barcelona (UAB), Spain, in 1988, where she is [19] T. Saito, K. Sato, H. Sato, and S. Matsuoka, ‘‘Energy-aware I/O opti- currently an Associate Professor with the Com- mization for checkpoint and restart on a NAND flash memory sys- tem,’’ in Proc. 3rd Workshop Fault-Tolerance HPC Extreme Scale, 2013, puter Architecture and Operating System Depart- pp. 41–48. ment. Her main research interests include parallel [20] B. Mills, R. E. Grant, K. B. Ferreira, and R. Riesen, ‘‘Evaluating energy computer architecture, parallel input/output, fault savings for checkpoint/restart,’’ in Proc. 1st Int. Workshop Energy Efficient tolerance in high-performance computing (HPC) Supercomput., 2013, pp. 6:1–6:8. systems, and tools to evaluate and predict the per- [21] D. Ibtesham, D. DeBonis, D. Arnold, and K. B. Ferreira, ‘‘Coarse- formance in parallel computers. grained energy modeling of rollback/recovery mechanisms,’’ in Proc. 44th Annu. IEEE/IFIP Int. Conf. Dependable Syst. Netw., Jun. 2014, E. LUQUE received the Doctor Honoris Causa pp. 708–713. from the National University of La Plata (UNLP), [22] A. Miyoshi, C. Lefurgy, E. van Hensbergen, R. Rajamony, and R. Rajkumar, ‘‘Critical power slope: Understanding the runtime effects Argentina, in 2018. He is currently an Emer- of frequency scaling,’’ in Proc. 16th Int. Conf. Supercomput., 2002, itus Professor with the Computer Architecture pp. 35–44. and Operating System Department, University Autònoma of Barcelona (UAB), Spain, where he is leading the HPC4EAS Research Group. He is also an Invited Professor at different universities in USA, Asia, Europe, and South America, a key note speaker in computer science conferences, and the leader in several research projects funded by the European Union (EU), the Spanish Government, and different companies. He has been the supervisor of 24 Ph.D. theses and has published more than 270 papers in technical journals and international conferences. His major research interests include simulation and optimization of emergency services in hospitals (smarter health services), performance and scalability prediction in high-performance M. MORÁN received the degree in computer computing (HPC) systems, efficient applications, and fault tolerance in science from Universidad Nacional del Com- parallel computers. He is also a member of the Editorial Board of various ahue, Neuquén, Argentina, in 1999. She is cur- technical journals. Recognitions for his research include the Distinguished rently pursuing the Ph.D. degree in computer Paper at the European Conference on Parallel Processing (Euro-Par 2006), science with the Universidad Nacional de La the Outstanding Paper Award at the International Conference on Compu- Plata, La Plata, Argentina. Her research interests tational Science (ICCS 2011), the Best Paper Award at the International include fault tolerance and energy consumption in Conference on Computational Science (ICCS 2012), and the Best Paper high-performance computing (HPC). Award at the Seventh International Conference on Advances in System Simulation (SIMUL 2015). VOLUME 7, 2019 71803