JCS&T Vol. 16 No. 2 November 2016 Using AWS EC2 as Test-Bed infrastructure in the I/O system configuration for HPC applications Pilar Gomez-Sanchez1 , Diego Encinas2 , Javier Panadero1 , Aprigio Bezerra3 , Sandra Mendez4 , Marcelo Naiouf2 , Armando De Giusti2 , Dolores Rexachs1 , and Emilio Luque1 1 Computer Architecture and Operating Systems Department,Universitat Aut` onoma de Barcelona,Campus UAB,Edifici Q,08193 Bellaterra(Barcelona), Spain. 2 Informatics Research Institute LIDI, National University of La Plata, La Plata, Buenos Aires, Argentina 3 Departamento de Ciˆencias Exatas e Tecnol´ogicas, Universidade Estadual de Santa Cruz, Ilh´eus, Bahia, Brasil 4 High Performance Systems Division, Leibniz Supercomputing Centre (LRZ), D-85748, Garching bei M¨ unchen, Germany. Abstract Cloud computing is gaining popularity in many areas, including HPC. Unlike on traditional HPC In recent years, the use of public cloud platforms as platforms, on a virtual cluster, users are their own infrastructure has been gaining popularity in many administrators, making it easy to change the I/O scientific areas and High Performance Computing parameters, such as the choice of the number of (HPC) is no exception. These kinds of platforms I/O nodes or the stripe size, in order to fit the I/O can be used by system administrators as Test-Bed requirements of the MPI application. Moreover, systems for evaluating and detecting performance as a main advantage, the changes carried out in inefficiencies in the I/O subsystem, and for taking the I/O subsystem do not have any influence on decisions about the configuration parameters that the production HPC system. have influence on the performance of an applica- In this paper, we discuss how a virtual HPC sys- tion, without compromising the performance of tem, deployed in cloud platform, can be employed the production HPC system. In this paper, we as a Test-Bed system to detect performance inef- propose a methodology to evaluate parallel appli- ficiencies and to tune the MPI application with cations by using virtual clusters as a test system. the I/O subsystem, so as to achieve high perfor- Our experimental validation indicates that virtual mance. In this way, system administrators can clusters are a quick and easy solution for system replicate its HPC platform as virtual clusters, in administrators, for analyzing the impact of the order to evaluate and take decisions about the I/O system on the I/O kernels of the parallel ap- configuration parameters that have an influence plications and for taking performance decisions in on the performance of an application, without a controlled environment. compromising overall system performance. Keywords: Cloud Computing, Parallel File Sys- To guide users in this evaluation, we propose a tem, PVFS2, MPI applications, methodology that strives on using a bounded time to evaluate the I/O subsystem. To achieve this 1 Introduction requirement, we have focused both on a reduced analysis time and a bounded time to deploy the Due to the complex interaction between the paral- virtual cluster. To achieve a reduced analysis time, lel I/O subsystem and the parallel message-passing the I/O patterns are analyzed to extract an I/O applications, the I/O subsystem can turn into a kernel of a parallel application, using a physical bottleneck in parallel systems, bringing about per- HPC system. Once this kernel is obtained, it is formance inefficiencies in the parallel applications. replicated in the virtual system using the IOR In order to use the system in an efficient way, benchmark or a synthetic program, to evaluate improving the performance, users and system ad- the I/O execution time and the bandwidth of ministrators have focused their efforts on tuning the application as performance indexes. In this the I/O subsystem with parallel applications. way, users can evaluate the performance of the Unfortunately, the high number of configuration application in the I/O subsystem in a quick and parameters, together with the fact that the right efficient way, in order to tune the configuration combination of parameters is highly dependent I/O system parameters. on the scientific application, results in making On the other hand, to deploy the I/O virtual it a complex task to modify any configuration subsystem, we have developed a plugin for the parameter in computation centers, where the HPC StarCluster tool [1], which allows us to deploy the system is in production 24/7. PVFS2 [2] file system quickly and automatically. 65 JCS&T Vol. 16 No. 2 November 2016 This paper is organized as follows. Section 2 the I/O system and evaluate them in a Cloud presents the related work. Section 3 introduces our environment. proposed methodology. In Section 4 we present the experimental evaluation, and in Section 5 the 3.1 Application Selection experimental validation. Finally, in the Section 6, we discuss the conclusions and future work. Recent advances in different scientific and engi- neering areas are generating large amounts of data that need to be stored and analyzed efficiently. 2 Related work These huge volumes of data must be processed in a satisfactory time to enable decision-making Although traditionally the research efforts in vir- and new discoveries. So we have a group of ap- tual HPC systems have been focused on evaluat- plications with different types of requirements. ing computation and communication, in recent Applications where the main requirement is not years, there has been increase of work focused the need for processing, but managing the huge on assessing the parallel I/O subsystems in cloud amounts of data within a reasonable time. Stor- environments. ing, transfering and processing large volumes of Liu et al. [3] demonstrate that the unique con- data become determining factors for systems that figurability advantage offered by public clouds handle these types of applications, and a challenge may bring opportunities for HPC applications to for high performance computing. achieve significant performance or cost improve- Reconciling these needs with requests from tra- ment. In particular, they illustrate the impact of ditional applications is a major challenge for ad- virtual cluster configurability by assessing the im- ministrators of multi-user clusters. In this type of pact of I/O system customization on the Amazon environment it is necessary to consider the sources EC2 system. There are other works which have of I/O contention when the cluster administrator been focused on optimizing cloud platform con- defines the job scheduling policy: CPU, Memory, figurations on the Amazon EC2 system. Gideon Storage I/O, Network I/O, System bus I/O and et al. [4] study the impact of different data shar- Blade chassis I/O. ing options for scientific workflows on Amazon It is necessary to examine whether the appli- EC2. Exposito et al. [5] evaluate the I/O storage cation triggers waiting times of I/O. When this subsystem on the Amazon EC2 CC platform to occurs, the make-span times tend to increase (mea- determine its suitability for scientific applications sured time from sending the first work to cluster with high I/O performance requirements . Vec- until the end of last work released). The cluster chiola et al. [6] execute a fMRI brain imaging administrator needs to assess where the bottle- workflow on Amazon EC2 using the Amazon Sim- neck is, if memory makes swap, if the I/O system ple Storage Service (S3) for storage, and analyzed collapses or both of these things. the cost varying the number of nodes. Moreover, there are some tools which are fo- 3.2 Analyzing the I/O properties cused on configurating the I/O subsystem in vir- tual clusters. Zhai et al. [7] propose ACIC (Au- HPC applications perform I/O operations by us- tomatic Cloud I/O Configurator), one of the first ing two common I/O strategies: serial and parallel. tools to optimize the I/O system for parallel appli- The second I/O strategy should be used to take cations in the cloud. Given a parallel application advantage of the performance capacity of the mod- to run on a given cloud platform, ACIC automat- ern HPC-I/O systems. Parallel I/O can be done ically searches for optimized I/O system configu- by using specialized I/O libraries or statements rations from many candidate settings. Herodotou provided by the different programming languages. et al. [8] propose Elastisizer, which can select the Understanding the behavior of the operation into proper cluster size and instance types for MapRe- file is crucial for detecting I/O bottlenecks and for duce workloads running in the cloud. proposing I/O tuning techniques. Our proposal differs from these approaches, be- In this section, we present the main I/O charac- cause our objective is to use the virtual cluster as teristics of parallel applications that are portable a Test-Bed platform, to take decisions about the to different I/O systems. These are selected to production HPC system. help the user to evaluate and understand the I/O behavior. Two main properties impact on the I/O perfor- 3 Proposed Methodology mance: the I/O concurrence degree and the data amount transferred into the file system. To rep- We propose a methodology that is composed of resent these two issues, we extract the following a set of steps that allow us to extract the I/O information: behavior of the parallel application and define its I/O kernel, define different configurations of • At parallel application level 66 JCS&T Vol. 16 No. 2 November 2016 – The number of files – The number of MPI processes – Storage capacity required – Data to be transferred • At file level – Access Mode: for p MPI processes a file can be accessed using a strided, sequen- tial or random pattern. – Access Type: Unique (a file per MPI process) or Shared (a shared File for all or a set of MPI processes). – Data access type: File can be write only (W ), read only(R) or write-read (W/R). – Number of I/O processes Figure 1: HPC I/O System – The count of concurrent accesses into the file system: the temporal pattern is extracted considering the time stamp of 3.4 Deploying the Test Environment opens and data access operations within on Cloud the different files. On Cloud, a HPC cluster can be created in similar – Data access operation properties: re- way to physical cluster because it offers individ- quest size and the frequency of the I/O ual HPC resources. The Cloud platforms provide operations. different types of instances that include a spe- HPC tracing and profiling tools can be applied cific number of CPU, memory, storage and the to extract the I/O properties. However, unlike networking performance. Furthermore, a cloud the compute performance analysis, the I/O perfor- environment usually provides image with a stack mance at file level is not a typical characteristic software to deployed Linux clusters. provided by the performance analysis tool. Due An automatic tool to create Linux Clusters is to this, we select the Darshan [9] tool to extract StarCluster [1], which helps users and system ad- the I/O profiling and apply our experience on the ministrators to create a virtual cluster using in- performance analysis to define the portable I/O stance of Amazon’s EC2 platform. We select this characteristics that represent the behavior of the tool to create a base HPC cluster. parallel application. Usually the HPC-I/O subsystem must be con- figured by the user in a Cloud environment. The 3.3 Replicating the I/O properties HPC-I/O system is represented as is shown in Figure 1. The I/O software stack comprises Sci- Once the application I/O behavior has been ob- entific Data Library (like HDF5 and NetCDF), tained, it is replicated by using a synthetic pro- I/O Middle-ware (like MPI-IO) and Parallel File gram or a benchmark with flexibility to reproduce System (like GPFS, Lustre and PVFS2) [11]. the I/O kernel of the parallel applications. In this point, we create an image on AWS EC2 An I/O benchmark that allows us to set up that implements the I/O software stack and this different I/O properties is IOR [10],which is de- is used to deploy a cluster using Starcluster tool. veloped to be used in tests on parallel file systems The user can select the type of instance and select in high-performance cluster. our image to have a fully functional HPC-I/O IOR can be used on any platform that provides system. an MPI implementation. It also offers the ability to test aggregate I/O rates through several typical 3.5 Configuring a Parallel Filesystem middle-ware libraries, including MPI-IO, HDF5, NetCDF and POSIX libraries. Parallel applications need a global file system to Input parameters allow us to vary the overall share the input and output data. Depending on size of the I/O operation, the individual size of the application I/O pattern, the type of file sys- each transfer, the mode of access to the file and tem can have a negative impact on the I/O per- if the data is accessed sequentially or randomly. formance. The most common global file systems These inputs can be used to reproduce I/O pat- in HPC cluster are NFS and parallel file systems terns of HPC applications. like Lustre, GPFS, PVFS2[12]. 67 JCS&T Vol. 16 No. 2 November 2016 (a) View of all MPI processes. (b) View for each MPI process Figure 2: Vampir Trace for ABYSS-P using 4 MPI processes. Test case for two small input files and k=25. User events in green, MPI events in red, and I/O events in dark green triangles and yellow bars. Starcluster tool deploys a cluster with NFS, HPC-IO systems. The metrics considered to ana- therefore, in order to use a parallel file system, lyze the results are data transfer rate, I/O time this must be installed and configured by the user. and I/O operations per second. A parallel appli- In this case, we have chosen PVFS2, which is an cation I/O kernel is replicated through the I/O open source file system developed to support effi- patterns identified and IOR benchmark on differ- cient read and write operations of large amounts ent I/O subsystems. of data. PVFS2 is designed as a client-server ar- chitecture where the server provides the storage and the client contains the access logic to the dis- 4 Experimental Evaluation tributed storage. Servers can be classified into In this section, we present the evaluation of the datafile (DF) and metadata servers (MDS). The parallel application ABySS (Assembly By Short former keep parts of logical files while the latter Sequences) [13], which reports slow running on the keep attributes of the logical file system objects CACAU [14] cluster of the NBCGIB (N´ ucleo de (files and directories in PVFS). Usually every I/O Biologia Computacional e Gest˜ao de Informa¸co˜es node is either a DF or an MDS. Biotecnol´ogicas, Brazil). The user provides us In general, parallel file systems have several with the input data and specific parameters to parameters that affect the performance. However, evaluate ABYSS. We perform an I/O pattern anal- in this work we focus on number of DFs, the stripe ysis in LRZ’s HPC systems [15] and define a set size and the placement of the DFs and MDS into of different configurations of the I/O system to be the architecture of the HPC cluster. The number implemented in a Cloud environment. of DF defines the file stripping and stripe size is the number of bytes written on one DF before cycling to the next DF. 4.1 ABySS Application Description ABySS is a parallelized sequence assembler. It 3.6 Analyzing Results is specialized in assembling large genomes from Parallel application performance can be affected short sequences, using a pair-end (forward and for several issues like memory, compute, communi- reverse) method. ABYSS is an example of a de cation or I/O into files. The user normally focuses novo assembling algorithm, where the construc- on obtaining the results in the least possible time. tion of a genome occurs in its pure form, without However, the administrator expects to have high consulting previously resolved references genomes. throughput at the HPC system level. The traditional short read de novo assemblers In this work, we focus our analysis on the I/O are single-threaded applications designed to run into files to identify and extract the I/O patterns on a single processor. However, these assemblers of parallel applications that are portable to other cannot process the total number of base pairs 68 JCS&T Vol. 16 No. 2 November 2016 sequenced by massively parallel sequencing plat- forms, such as the Illumina, Inc. Genome Analyzer or Applied Biosystems SOLiD System, due to the computation time and memory constraint limits. One advantage of the ABySS compared with other short read de novo assemblers is that it al- lows a distribution of the assembly algorithm in a parallel way (master/worker architecture). It is possible because it uses a distributed representa- tion of a de Bruijn graph [16]. This kind of graph is a simple way to represent DNA sequences where every read corresponds to a vertex, and two ver- tices are connected by an edge if the corresponding reads overlap. The main feature of a de Bruijn graph is to allow the representation of sequence regions that repeat as a unique edge. Thus, the Figure 3: Access mode for ABYSS-P using 4 MPI problem of fragment assembly is reduced to find a processes. Sequential: An I/O operation issued path visiting every edge of the graph exactly once. at an offset greater than where the previous I/O The ABySS algorithm proceeds in two phases. operation ended. Consecutive: An I/O operation The first phase loads short read sequences, break- issued at the offset immediately following the end ing each into k-mers of length k. Then, it finds of the previous I/O operation. adjacent (overlapping) k-mers and it removes k- mers resulting from read errors. In the second phase, mate-pair information is used to remove and MPI events (red bars), green triangles rep- variant sequences (bubbles) and to generate con- resent the file open and close, and the red/green tiguous sequences (contigs). triangles represent multiple I/O event at the same However, there are reports that indicate prob- time interval. In the first I/O phase, the pro- lems in ABySS algorithm scalability. The costly cesses with rank 0 and 1 perform I/O activities; communication and load-balancing inefficiencies and in the second phase (at running end) all MPI emerge from larger scale. It is because the algo- processes write to output files. rithm executes master-worker model with asyn- To obtain detailed information about the I/O chronous message/work complexes queues, and pattern in each phase we analyze ABySS using uses MPI P2P and collective communication that Darshan tool. A total of eleven files are used varies by execution stage. Furthermore, there by ABYSS-P that corresponds to 2 files per each are extreme run-to-run execution time variations MPI process, two read files and 1 output file. I/O (which is likely to be mostly due to file I/O). patterns are represented by the sequential access mode (see Figure 3) with mainly consecutive ac- cesses for read operations, small request size (see 4.2 Analyzing the ABySS I/O proper- ties Figure 4(b)), read and seeks operations (see Fig- ure 4(a)) that correspond to operations into the Considering the ABYSS-P description a first anal- input files. The I/O is serial by using the C I/O ysis is performed to identify the I/O phases. We library. select VampirTrace to start with a small test and Figure 4(c) depicts the temporal pattern for Darshan for larger number of MPI processes and ABYSS-P where we can observe that only two I/O workload. VampirTrace tool allows us to first processes perform read operations into observe the full behavior of the application and reads1.fastq and reads2.fastq files. Write op- Darshan tool extracts the I/O patterns. erations take a short time, so for this reason we We start the analysis of ABySS-P for a small cannot observe them in the timeline. test by using 4 MPI processes, to assemble Once the pattern for this case has been analysed, paired reads in two files named reads1.fastq we need to define the general phases and the I/O and reads2.fastq into contigs in a file named pattern that will be applied to evaluate ABYSS-P test-bubbles.fa; and k=25. for real input files and number of MPI processes. The test case is shown in Figure 2. This small To do this, we select the Darshan tool due to its case allows us to identify two I/O phases at the be- light overhead and small trace file size compared ginning of the running and at the ending. Figure to other tracing tools. 2(a) shows the timeline for the 4 MPI processes. ABYSS-P uses a total of np ∗ 2 write files, two In Figure 2(b) a trace view for each process is de- input files and 1 output file, where np is the num- picted, where row 1 corresponds to user functions ber of MPI processes for the parallel execution. and row 2 represents the I/O events (yellow bars) Input files are read in a first phase where only the 69 JCS&T Vol. 16 No. 2 November 2016 (a) Operation counts (b) Request Size (c) Temporal Pattern ABYSS-P using 4 MPI processes. The y-Axis represents the MPI ranks and the x-Axis the time in hours:minute:seconds. A total of 11 files was open but only the read files are shown because are representing the I/O with more impact on the performance. Figure 4: ABYSS-P’s I/O characteristics reported by Darshan tool for 4 MPI processes and test input files. Table 1: Read Phase for ABYSS-P by using np MPI processes for two real input files. Two files of 17 GiB are read by two I/O processes (I/O proc) that corresponds to rank 0 and rank 1. Request size (rs) and access mode are independent of the input files. np rs(B) I/O proc File per #iop Data x I/O proc Phase Data I/O proc 24 8191 2 1 2135744 x 2 17 GiB 33 GiB 32 8191 2 1 2135744 x 2 17 GiB 33 GiB 64 8191 2 1 2135744 x 2 17 GiB 33 GiB 128 8191 2 1 2135744 x 2 17 GiB 33 GiB 512 8191 2 1 2135744 x 2 17 GiB 33 GiB 70 JCS&T Vol. 16 No. 2 November 2016 rank 0 and 1 read a file each one independently of Pabplug PVFS2 number of MPI processes. The Read phase moves StarCluster config Plugin more than 90% of the data and I/O operations Virtual HPC Cluster of ABYSS-P. Table 1 shows detailed information with PVFS2 by using different number of MPI processes only focus on read phase. Figure 5: Integrating the pvfs2 plugin with Star- Cluster 4.3 Replicating the ABYSS-P’s I/O properties Table 2: Description of the CACAU cluster Components CACAU We design a set of experiments to replicate the CACAU0 CACAU1 ABYSS-P’s I/O properties by using the IOR Compute Nodes 20 3 benchmark (IOR 2.10.3) and only focus on the Processor 2 2 read phase showed in Table 1. (per node) Intel Xeon Intel Xeon IOR benchmark is configured as follow: E5430 E5-2400 (2.66GHz) (2.40 GHz) mpirun -np NP ./IOR -r -N 1 -a POSIX CPU cores 8 12 -b 17496014848 -t 8192 -k (per node) Where: GPU cores - 2498 (per node) • NP represents the number of MPI processes RAM Memory 16 GB 48 GB to evaluate. (per node) Local FileSystem ext4 ext4 • -r selects only read test because we are only Capacity of Local 160 GB 500 GB analyzing the ABYSS-P’s read phase. Storage (per node) • -N sets up the number of I/O processes, in Device Type of Lo- HDD HDD this case 1. cal Storage MPI Library OpenMPI 1.10.2 • -a sets up the I/O library. Operating System RedHat Linux Server 6.3 Global FileSystem NFS v4 • -b defines the contiguous data in bytes to Capacity of Global 2 TB read for each I/O process that corresponds FileSystem to the total size of an input file of ABYSS-P. Storage Total 8 TB • -t defines the request size in bytes that in Storage Network 4 Gbps Ethernet this case is 8192 because IOR does not allows the value 8191. • -k indicates that the read file has not be Using the regular mode, users only have the removed when the test finishes. possibility to specify the I/O nodes and the meta- node, the rest of pvfs2 parameters are set up by To replicate the I/O concurrency into two in- default. On the other hand, using the expert mode, put files, two concurrent executions of IOR with users have full control of the plugin. In this way, the previous configuration are run on different users can specify all configuration parameters of configurations of the I/O system. PVFS2 explained in [17]. Moreover, users can specify the placement of the MDS and DFs into 4.4 Deploying a virtual cluster HPC cluster architecture. In this case we define two placements for the DFs: 1) a DF defined In order to deploy the virtual cluster we use the on each compute node of the cluster or, 2) DFs StarCluster tool. As currently StarCluster does defined on independent nodes, which means that not deploy a HPC cluster with a parallel filesystem, these will only be I/O nodes. a plugin has been developed to be integrated with the StarCluster tool, which allows us to deploy the PVFS2 file system automatically. 4.5 Configuring the different I/O sub- This plugin has been integrated using the stan- systems on Cloud dard plugin method provided by StarCluster. To A set of experiments is designed by using the IOR configure the PVFS2 file system, the plugin pro- configuration defined in Section 4.3, and these ex- vides a configuration file, as is shown in Figure 5. periments are executed in the following scenarios: In this configuration file, users can choose between two installation ways: regular and expert mode. • NFS: i) Mapping an I/O process in differ- 71 JCS&T Vol. 16 No. 2 November 2016 Table 3: Characteristics of the Amazon’s Instances Selected Instances Processor vCPUs RAM Storage Network $xHora (GiB) (GB) t2.medium Intel Xeon (Turbo up 2 4 only EBS Low to Moderate 0.052 to 3.3GHz) c3.2xlarge Intel Xeon E5-2680 v2 8 15 2x80 (SSD) High 0.42 (Ivy Bridge) ent compute nodes and ii) Mapping two I/O label 1PxCN, and 2PxCN for the mapping the processes on the same compute node. two I/O processes on the same compute node. Figure 6(b) presents results for the c3.2xlarge • PVFS2: i) Mapping an I/O process in differ- instance. In this case, the configuration PVFS- ent compute nodes and ii) Mapping two I/O 1DFx1F means that each input file is striped to 1 processes on the same compute node. Each DF but different DF for each one. one on different data layout: The results show that NFS reports more data – Each input file placed in 1 different DF. transfer rate than PVFS. This result is the same for the different instances and it can be observed – Each input file stripped into different that the bandwidth does not improve using more numbers of DFs. DFs. The I/O processes mapping is considered be- On the NFS file system, the mapping is im- cause each process reads a 17 GB file, which has pacting on data transfer rate and this is more an impact on the memory utilization and it can significant for VCC with compute nodes with 4 impact on the application performance, especially GiB RAM. On PVFS2, we can observe an im- for compute nodes with less than 33 GB of RAM. provement for 1PxCN. The data layout 1DFx1F To select the instance type for the testing, we reports more performance than other PVFS2 I/O try to use an instance with similar characteristics configurations. to the compute nodes of the CACAU cluster. The Discussion ABYSS-P is a parallel application CACAU features are shown in Table 2, which that is reading from two files by using an I/O is composed of two segments. We focus on the process for each file. This behavior is the same in- CACAU0 segment. dependently of the number of MPI processes. This We create clusters with six nodes: 1 master I/O pattern cannot take advantage of the perfor- and five compute nodes. The AWS EC2 instance mance capacity provided by a parallel filesystem similar to the CACAU0 segment is the c3.2xlarge like PVFS2. The results in Figure 6 show this (see Table 3). We also select the t2.medium (see problem. Although we increase the number of Table 3) to check it is possible to demonstrate the DFs, the I/O performance presents similar values. hypothesis working with instances that have lower Furthermore, the small request size impacts on features. Two Virtual Clusters are configured with the performance of PVFS2 when only one process these two types of instances that are described in is carrying out I/O. We can observe that NFS can Table 4. provide an acceptable I/O performance. However, For each VCC we use 6 instances to evaluate we must to take care in this point because the NFS and PVFS2 file systems. For PVFS2 we use NFS filesystem for this kind of application should a maximum of 3 DFs be able to evaluate the two be different to the NFS for home user accounts. DFs placement on the HPC-VCC’s architecture. Moreover, we have not evaluated the impact that the stripe size and the number of MDS can 4.6 Analyzing the I/O behavior on dif- have. Other parallel file systems like Lustre or ferent scenarios GPFS show more bandwidth for this kind of I/O We evaluate the I/O kernel of the ABYSS-P in pattern. However, in this work, we select PVFS2 the scenarios defined in section 4.5 and we obtain for the simple installation and free distribution. different performance metrics on PVFS2 and NFS. During the process of the I/O pattern extraction In this case, we present the result of data transfer needed to define the I/O kernel of ABYSS-P, we rate and I/O operations per second (IOPs). Figure have observed that the I/O pattern is a problem 6(a) shows results by using instance t2.medium. for the scalability. Due to the fact only two pro- The x-axis represents the different configurations cesses are reading the input files and sending read for the file system, PVFS2 by using different num- data to the rest of the processes, this is clearly ber of DFs and NFS. The mapping an I/O process an inefficient I/O pattern and an obstacle for the in different compute nodes is represented by the ABYSS-P scalability. 72 JCS&T Vol. 16 No. 2 November 2016 Table 4: Descriptive Characteristics of the Virtual Clusters on cloud (VCC) configured equal for the experiments. I/O components VCC 1 VCC 2 VCC 3 VCC 4 Instance t2.medium t2.medium c3.2xlarge c3.2xlarge Number of Instances 6 6 6 6 Temporal Storage - - Ephemeral Ephemeral Persistent Storage EBS EBS EBS EBS Temporal Device - - SSD SSD Persistent Device SSD(GP SSD(GP SSD(GP SSD(GP 192/3000) 300/3000) 192/3000) 300/3000) Capacity of Persistent Storage 100GB 64GB 100GB 64GB File system Local ext4 ext4 ext4 ext4 File system Global NFS PVFS2 NFS PVFS2 (2.8.8) (2.8.8) Parallel Storage Capacity - 500GB - 400GB Number of data servers - 5 - 5 Number of Metadata Server - 1 - 1 Stripe Size - 64KB - 64KB MPI library mpich-3.2 mpich-3.2 mpich-3.2 mpich-3.2 (a) Instance t2.medium (b) Instance c3.2xlarge Figure 6: Data transfer rate and IOPs for IOR configured to ABYSS-P’s I/O Kernel. 73 JCS&T Vol. 16 No. 2 November 2016 Table 5: Description of the CAPITA cluster Components CAPITA CAP0 CAP1 Compute Nodes 13 13 CPU cores (per node) 4 4 RAM Memory 16GB 16GB Capacity of Local Storage (per node) 46GB 46GB Device Type of Local Storage SSD SSD Local Filesystem Linux ext3 Linux ext3 Global Filesys- NFS PVFS2 tem (2.8.7) Capacity of Global FileSys- 46GB 175GB tem MPI-Library mpich2 mpich2 (1.5) (1.5) Number of data - 4 Figure 7: Data transfer rate and IOPs for IOR servers configured to ABYSS-P’s I/O Kernel. Results Number of Meta- - 1 obtained in CAPITA cluster. data Server Stripe Size - 64K Network 1 Gbps 1 Gbps Ethernet Ethernet 6 Conclusion In this paper, we have analyzed the real parallel application ABYSS-P on different configurations 5 Experimental Validation of the I/O subsystem without changing the I/O system of CACAU physical cluster. In this section, we validate the results obtained In Cloud environments the time consumed by in a Cloud environment by replicating the experi- the instances is fundamental because we pay for ments defined in Section 4.5, in the physical cluster the time, for this reason we only evaluate the CAPITA. This is described in Table 5, the NFS ABYSS-P’s I/O kernel. This is obtained in a and PVFS2 file spaces are configured using SSD physical cluster and it can be executed on different devices. The NFS file space is a special partition HPC systems. only used for intensive I/O that is different to the NFS used as HOME file space. Due to phys- Our work shows that the Cloud allows different ical nodes being more comparable to instances I/O configurations to be deployed automatically C3.2xlarge, we focus on the validation comparing and their influence on the I/O performance of the the CAPITA results with VCCs experiments that parallel application to be evaluated quickly. use instances C3.2xlarge. From this evaluation we can suggest the user In previous IOR experiments on a Cloud envi- tries to modify the I/O pattern because it will not ronment (see Section 4.6), we have shown that the be able to take advantage of a parallel file system. PVFS2 provides less I/O performance than NFS. On other hand, the administrator must try to These results are also observed in the CAPITA allocate the application on compute nodes with cluster. Experimental results are shown in Figure more memory, especially for the I/O processes 7, where the same behavior for the ABYSS’s I/O that usually need more memory for the input files. kernel both in VCCs (see Figure 6(b)) as in the CAPITA cluster can be observed. Therefore, we Experimental results showed that the parallel can say that the Cloud environment can be used file system selected using different configurations as Test-Bed infrastructure as long as the virtual has less performance than NFS. The mapping cluster is designed considering the main compo- an I/O process per compute node on a NFS file nents of the physical cluster and the requirements system was the most suitable configuration for of the application’s I/O kernel. this kind of pattern. 74 JCS&T Vol. 16 No. 2 November 2016 Acknowledgements of scientific applications,” in 2009 10th Inter- national Symposium on Pervasive Systems, This research has been supported by the MINECO Algorithms, and Networks, pp. 4–16, 2009. (MICINN) Spain under contract TIN2014-53172- [7] J. Zhai, M. Liu, Y. Jin, X. Ma, and W. Chen, P and partially supported by the CloudMas as “Automatic cloud i/o configurator for i/o in- Government Competency of AMAZON Web Ser- tensive parallel applications,” IEEE Trans- vices (AWS). The research position of the PhD actions on Parallel and Distributed Systems, student P. Gomez has been funded by a research vol. 26, no. 12, pp. 3275–3288, 2015. collaboration agreement, with the ”Fundaci´on Es- cuelas Universitarias Gimbernat”. The authors [8] H. Herodotou, F. Dong, and S. Babu, “No one thankfully acknowledge the resources provided by (cluster) size fits all: Automatic cluster sizing the Centre of Supercomputing of Galicia (CESGA, for data-intensive analytics,” in Proceedings Spain) and the Leibniz Supercomputing Centre of the 2Nd ACM Symposium on Cloud Com- (LRZ, Germany). puting, SOCC ’11, pp. 18:1–18:14, 2011. We thank Eduardo Almeida Costa, PhD student of the Program in Biology and Biotechnology of [9] P. Carns, K. Harms, W. Allcock, C. Bacon, Microorganisms at the State University of Santa S. Lang, R. Latham, and R. Ross, “Under- Cruz (Brazil), who provided us with the input standing and improving computational sci- data for the ABySS application. We would also ence storage access through continuous char- like to show our gratitude to Pawel Wichary for acterization,” Trans. Storage, vol. 7, pp. 8:1– collaborating in the implementation of the PVFS2 8:26, Oct. 2011. plugin. [10] W. Loewe, T. MacLarty, and M. C, IOR References Benchmark, 2012. Accessed: 2016-05-14. [1] “Starcluster by mit.” [11] R. Ross, R. Thakur, and A. Choudhary, http://http://star.mit.edu/. Accessed: “Achievements and challenges for i/o in com- 2016-05-02. putational science,” Journal of Physics: Con- ference Series, vol. 16, no. 1, pp. 501+, 2005. [2] N. Miller, R. Latham, R. Ross, and P. Carns, “improving cluster performance with pvfs2,” [12] Prabhat and Q. Koziol, High Performance ClusterWorld Magazine, vol. 2, no. 4, 2004. Parallel I/O. Chapman & Hall/CRC Com- putational Science, 1 ed., 2014. [3] M. Liu, J. Zhai, Y. Zhai, X. Ma, and W. Chen, “One optimized i/o configuration per hpc ap- [13] J. T. Simpson, K. Wong, S. D. Jackman, J. E. plication: Leveraging the configurability of Schein, S. J. Jones, and I. Birol, “Abyss: a cloud,” in Proceedings of the Second Asia- parallel assembler for short read sequence Pacific Workshop on Systems, APSys ’11, data,” Genome research, vol. 19, no. 6, pp. 15:1–15:5, 2011. pp. 1117–1123, 2009. [4] G. Juve, E. Deelman, K. Vahi, G. Mehta, [14] CACAU, “N´ ucleo de Biologia Computacional B. Berriman, B. P. Berman, and P. Maechling, e Gest˜ao de Informa¸c˜oes Biotecnol´ogicas,” “Data sharing options for scientific workflows tech. rep., Universidade Estadual de Santa on amazon ec2,” in 2010 ACM/IEEE Inter- Cruz, 2016. national Conference for High Performance Computing, Networking, Storage and Analy- [15] LRZ, “Leibniz Supercomputing Centre,” sis, pp. 1–9, 2010. tech. rep., Bayerischen Akademie der Wis- [5] R. R. Exp´osito, G. L. Taboada, S. Ramos, senschaften, 2016. J. Gonz´ alez-Dom´ınguez, J. Touri˜ no, and R. Doallo, “Analysis of i/o performance on [16] P. A. Pevzner, H. Tang, and M. S. Waterman, an amazon ec2 cluster compute and high i/o “An eulerian path approach to dna fragment platform,” Journal of grid computing, vol. 11, assembly,” Proc Natl Acad Sci USA, vol. 98, no. 4, pp. 613–631, 2013. pp. 9748–53, Aug. 2001. [6] C. Vecchiola, S. Pandey, and R. Buyya, [17] PVFS2, “Config File Description,” tech. rep., “High-performance cloud computing: A view PVFS2, 2016. 75