sensors Article Packet Flow Capacity Autonomous Operation Based on Reinforcement Learning Sima Barzegar , Marc Ruiz and Luis Velasco * Optical Communications Group, Universitat Politècnica de Catalunya, 08034 Barcelona, Spain; sima.barzegar@upc.edu (S.B.); marc.ruiz-ramirez@upc.edu (M.R.) * Correspondence: luis.velasco@upc.edu Abstract: As the dynamicity of the traffic increases, the need for self-network operation becomes more evident. One of the solutions that might bring cost savings to network operators is the dynamic capacity management of large packet flows, especially in the context of packet over optical networks. Machine Learning, particularly Reinforcement Learning, seems to be an enabler for autonomicity as a result of its inherent capacity to learn from experience. However, precisely because of that, RL methods might not be able to provide the required performance (e.g., delay, packet loss, and capacity overprovisioning) when managing the capacity of packet flows, until they learn the optimal policy. In view of that, we propose a management lifecycle with three phases: (i) a self-tuned threshold- based approach operating just after the packet flow is set up and until enough data on the traffic characteristics are available; (ii) an RL operation based on models pre-trained with a generic traffic profile; and (iii) an RL operation with models trained for real traffic. Exhaustive simulation results  confirm the poor performance of RL algorithms until the optimal policy is learnt and when traffic  characteristics change over time, which prevents deploying such methods in operators’ networks. Citation: Barzegar, S.; Ruiz, M.; In contrast, the proposed lifecycle outperforms benchmarking approaches, achieving noticeable Velasco, L. Packet Flow Capacity performance from the beginning of operation while showing robustness against traffic changes. Autonomous Operation Based on Reinforcement Learning. Sensors 2021, Keywords: reinforcement learning; autonomous network operation; offline/online learning 21, 8306. https://doi.org/10.3390/ s21248306 Academic Editors: Ramon Aparicio-Pardo 1. Introduction and Peter Han Joo Chong Autonomous network operation evolves from software-defined networking (SDN) [1] and promises to reduce operational expenditures by implementing closed loops based Received: 7 November 2021 on data analytics [2,3]. Such control loops can be put into practice based on policies that Accepted: 10 December 2021 specify the action to be taken under some circumstance (policy-based management), e.g., Published: 12 December 2021 allocate the capacity of a packet flow so that the ratio traffic volume over capacity is under 80%. In packet flows, this ratio is related to the average delay that the packets in the flow Publisher’s Note: MDPI stays neutral will experience because of queuing, and thus to the Service Level Agreement (SLA) in the with regard to jurisdictional claims in case that the packet flow is related to some customer connection. Although such policies published maps and institutional affil- can be modified, they are purely reactive. Under high traffic variations, they might entail iations. either poor Quality of Service (QoS) (e.g., high delay or even traffic loss) and SLA breaches or poor resource utilization, which in both cases represent large costs for network operators. Note that policy-based management does not define the desired performance and thus, agents implementing those policies are unable to learn the best actions to be taken. Another Copyright: © 2021 by the authors. approach for network automation is Intent-Based Networking (IBN) [4,5]. IBN allows the Licensee MDPI, Basel, Switzerland. definition of operational objectives that a network entity, e.g., a traffic flow, has to meet This article is an open access article without specifying how to meet them. IBN implements and enforces those objectives, often distributed under the terms and with the help of Machine Learning (ML) [6]. Although many ML techniques could be conditions of the Creative Commons potentially applied for the autonomous capacity operation of traffic flows, in this paper, Attribution (CC BY) license (https:// we rely on Reinforcement Learning (RL) [7]. We consider several RL methods of different creativecommons.org/licenses/by/ complexity and analyze their performance, in particular: (i) Q-learning; (ii) dueling double 4.0/). Sensors 2021, 21, 8306. https://doi.org/10.3390/s21248306 https://www.mdpi.com/journal/sensors Sensors 2021, 21, 8306 2 of 24 deep Q-learning (denoted D3QN) [8], and (iii) Twin Delayed Deep Deterministic Policy Gradient (TD3) [9]. With the deployment of network slicing [10] and the support to time-sensitive ap- plications [11], the flow capacity autonomous operation (hereafter referred to as CRUX) focuses on answering a major problem that network operators are facing nowadays: how to allocate the right capacity to every traffic flow, so as to provide the desired QoS (e.g., by preventing traffic loss and ensuring a given maximum average delay), while minimizing overprovisioning (i.e., capacity–traffic). The next subsections provide the needed background, review the related works, and present the main contributions of this work. 1.1. Background on RL methods RL considers the paradigm of an agent interacting with its environment for learning reward-maximizing behavior (learning agent). At every discrete time step t, with a given state s, the agent selects action a with regard to a policy, and it receives from the environment a reward r and the new state s’. The objective is to find the optimal policy that maximizes the expected return. The simplest RL is Q-learning [7], which is a model-free discrete RL method that is able to learn the optimal policy represented by a Q-table of pairs , each containing a q value. Being at state s, an action a is taken—either the one corresponding to the highest q value, or chosen randomly. After the action is implemented, it is evaluated by receiving the new state, s’, and the gained reward r from the environment. The agent then updates the corresponding q value in the Q-table. Q-learning works efficiently for problems where both states and actions are discrete and finite. However, it suffers from two main problems: it introduces overestimation, which leads to suboptimal policies, and the Q-table grows with the number of states. Deep Q-learning (DQN) substitutes the Q-table by a feed-forward deep neural network (DNN) that receives a continuous representation of the state and returns the expected q value for each discrete action [7]. Because the DNN tends to make learning unstable, countermeasures need to be taken. The most extended measure is to keep a replay buffer storing the last experiences, i.e., tuple that is used to retrain the DNN. In addition, double DQN [12] uses two different DNNs (learning and target) to avoid overestimation, which happens when a non-optimal action is quickly biased (due to noise or exploration) with a high q value that makes it preferably selected. Thus, the learning model is updated using the q values retrieved from the target DNN, which is just a simple copy of the learning model that is periodically updated. Finally, dueling double DQN (named D3QN in this paper) [8] uses two different estimators to compute the q value of a pair : (i) the value estimator, which can be intuitively seen as the average q value of any action taken at state s; and (ii) the advantage estimator, which is the specific state action-dependent component. The sum of both value and advantage components returns expected q values. DQN-based methods assume a finite discrete action space. If continuous state and action spaces are required, other approaches such as Actor–Critic methods [9] can be used. The main idea behind Actor–Critic methods is that two different types of models are trained separately: (i) actors, who are in charge of computing actions based on states, and (ii) critics, who are in charge of evaluating the actions taken by actors, i.e., to compute q values. Both actor and critic models can be implemented by means of DNNs. Among all different Actor– Critic methods, the Twin Delayed Deep Deterministic Policy Gradient (TD3) method [9] considers one single actor and two different critic models, where the minimum value from the two critics is used to learn, aiming at reducing overestimation. 1.2. Related Work Several works have proposed the application of RL algorithms for autonomous net- work operation. In the context of optical networking, the authors in [13] proposed the use of Actor–Critic-based algorithms running in the SDN controller for dynamic lightpath pro- Sensors 2021, 21, 8306 3 of 24 visioning. They showed that changes in the traffic profile impact the obtained performance. The authors of [14] extend their work from [13] to multi-domain scenarios where multiple RL agents improve their learning by transferring knowledge. The authors of [15] studied the application of several deep RL algorithms (including DQN) and reward functions to increase network availability in SDN-controlled wide area networks. In the context of IBN, the work in [16] applied Q-learning to the autonomous operation of optical connections based on subcarrier multiplexing. Their RL algorithm, running in the optical transponder, activated and deactivated subcarriers to adapt the capacity of the optical connection to the upper layer packet traffic. The authors of [17] proposed an IBN solution based on Q-learning, which demonstrated the ability to learn the optimal policy based on the speci- fied operational objectives related to the allocated capacity or the delay. Finally, the work in [18] proposed an RL algorithm for autonomous bandwidth allocation in the context of multilayer optical networks. They proposed a centralized algorithm running in the SDN controller that supervises the network performance. When the network performance degrades, the centralized algorithm tunes parameters in the RL algorithms. One of the key issues in the previous works is the time needed to learn optimal policies, as exploration entails low-reward decision making (i.e., far from optimal operation), as shown in [13]. In view of this, the performance of RL methods is typically evaluated after some training phase, i.e., when reward achieves a stationary behavior. However, a subject that is poorly or not even considered in the literature is when RL algorithms need to operate before they are properly trained. Note that this happens in our case, as the actual characteristics of the traffic flow are unknown until it is provisioned. Moreover, it is not realistic to assume, in general, the same conditions during training and operation phases, due to mid/long-term traffic evolution, which makes it difficult to reproduce highly accurate operation conditions during training. To solve these issues, the authors of [19] proposed a general learning lifecycle that included both offline training (e.g., in a sandbox domain [20]) and online learning, in the context of supervised ML. The objective of that work was to accelerate autonomous operation by deploying accurate models that are firstly trained offline and fine-tuned while in operation, thus adapting pre-trained models to actual in-field operation conditions. 1.3. Contributions In this paper, we apply the main lessons learnt from the previous works to IBN agents based on RL. We assume that a packet flow (alternatively, referred as traffic flow or simply flow) conveying traffic with unknown characteristics is established and the allocated capacity needs to be set to ensure the required QoS (from the set-up time), while minimizing resource utilization. To this end, a policy-based management is used at the set-up time to start operating the capacity of the flow; meanwhile, traffic measurements are collected to characterize the traffic. Note that policy-based operation can be highly reliable, as it is based on specific rules that can be defined and understood by human operators. However, such an operation usually obtains poor resource utilization. Therefore, it would be useful to substitute policy-based operation by an RL model as soon as possible. To that end, pre-trained generally applicable models for the partly observed traffic characteristics are loaded and the RL algorithm starts operating. A per-flow algorithm supervises the performance of the RL algorithm and tunes model parameters to ensure the required QoS. Once enough traffic measurements are available, offline training is carried out in a sandbox domain to produce a specific model well-suited for that particular traffic flow, which then substitutes the generic one. Since the traffic characteristics can change over time, analysis must be continuously performed to detect them and change the operating model when needed. By iterating that cycle, the proposed RL agent will be able to adapt to traffic changes that would otherwise degrade performance. The rest of the paper is organized as follows. The pros and cons of applying RL to real network operation are highlighted in Section 2, which include poor performance during the initial set-up and during changes in the traffic flow. The CRUX problem and the proposed ing the initial set-up and during changes in the traffic flow. The CRUX problem proposed approach are as follows: it targets operating a traffic flow and ensurin quired QoS from set-up time without previous knowledge of the traffic charac Sensors 2021, 21, 8306 The motivation behind the different phases of operation and the proposed 4 of 24 lifecyc sented, where pre-trained models and offline–online RL cycles aim at solving th fied issues. The CRUX problem is formally defined in Section 3. Several RL app can be used approach are asto solve itCRUX, follows: and eacha one targets operating requires traffic flow anddifferent ensuring thesettings requiredofQoS states and from set-up time without previous knowledge of the traffic characteristics. A methodological approach is provided to model the problem with different R The motivation behind the different phases of operation and the proposed lifecycle is presented, where rithms, and how the QoS targets are related to parameters in the reward function pre-trained models and offline–online RL cycles aim at solving the identified issues. The 4 presents CRUX problemthe algorithms is formally definedthat analyze in Section the RL 3. Several traffic characteristics, approaches can be used tosupervise solve the mance, CRUX, and and make each decisions one requires regarding different the settings of model states andand parameters actions. to be used for op A methodological approach is provided The discussion istosupported model the problem by thewith different results RL algorithms, presented and how in Section the QoS Sectio 5. Finally, targets are related to parameters in the reward function. Section 4 presents the algorithms cludes the paper. that analyze the traffic characteristics, supervise the performance, and make decisions regarding the model and parameters to be used for operation. The discussion is supported 2. the by The Flowpresented results Capacity Autonomous in Section 5. Finally,Operation (CRUX) Section 6 concludes theProblem paper. 2.1. Flow Capacity Autonomous Operation 2. The Flow Capacity Autonomous Operation (CRUX) Problem LetCapacity 2.1. Flow us start by analyzing Autonomous the result of the autonomous Operation capacity operation of flow.LetAusflow startmanager might by analyzing collect the result monitoring of the autonomous data fromoperation capacity the network and expo of a traffic flow. A flow manager might collect monitoring data from the network and expose interface, so an RL algorithm can take action. For illustrative purposes, Figure 1 some interface, so an RL algorithm can take action. For illustrative purposes, Figure 1 shows a typical RL framework [21], where the learning agent is separated into two differen typical RL framework [21], where the learning agent is separated into two different blocks, thelearner the learnerandand the agent. the agent. RL-based Operation Learner Online Learner Buffer Reward Model Action Updates Agent Models Action State Environment State function Reward function Traffic, Required Capacity Capacity Flow Manager Monitoring Capacity Flow Figure Figure1. Flow capacity 1. Flow autonomous capacity operation. operation. autonomous RL framework RLwith learner, agent, framework withand learner, environment. agent, and ment. Let us assume that the monitoring data include the byte count since the last monitoring sample (amount of traffic) and the actual capacity allocated to the flow. The actions to Letare be taken usrelated assume that to the the monitoring actual datatoinclude capacity allocated the flow,the bytecan which count since the last m be increased or decreased ing sampleas(amount needed with some granularity of traffic) to meetcapacity and the actual the required QoS. Fortoinstance, allocated the flow.a The a customer connection can manage the capacity of the flow with granularity be taken are related to the actual capacity allocated to the flow, which can be incr 1 Gb/s by configuring some packet node, whereas in a virtual link supported by the optical layer, the decreased as needed with some granularity to meet the required QoS. For instanc capacity can be increased/decreased by establishing or tearing down parallel lightpaths, tomer each with connection a capacity ofcan 100manage s Gb/s. Itthe capacity seems clear thatofthe thetime flow to with changegranularity the capacity 1is Gb/s by uring also someranging different, packetfrom node, whereas seconds in a virtual to minutes. link supported The RL algorithm by the should then optical decide the layer capacity pacity to be be can allocated to the flow to absorb increased/decreased byvariation in the traffic establishing from onedown or tearing monitoring parallel lig sample to the next, plus the time to increase the allocated capacity, with the objective to each with a capacity of 100 s Gb/s. It seems clear that the time to change the ca avoid any traffic loss and ensure some additional QoS metric. Then, the traffic variation also different, ranging from seconds to minutes. The RL algorithm should then decide the capacity to be allocated to the flow to absorb variation in the traffic from one monitoring sample to the next, plus the time to increase the allocated capacity, with the objective to Sensors 2021, 21, 8306 5 of 24 avoid any traffic loss and ensure some additional QoS metric. Then, the traffic variation becomes a major feature for a flow, together with the traffic pattern, i.e., the evolution of the mean traffic with time. becomes a major feature This approach for a flow, can provide excellent with the traffic once togetherperformance pattern, thei.e., the evolution policies of thetraffic that avoid mean traffic with time. losses meet the desired QoS, and minimize overprovisioning are learned; however, online learning This approach of such can provide policies requiresexcellent time. Inperformance addition, thereoncearetheseveral policiesissues that avoid that traffic can impact losses meet the desired QoS, and minimize overprovisioning are learned; however, online the aforementioned online learning performance, e.g., (1) changes in traffic variability learning of such policies requires time. In addition, there are several issues that can impact might produce loss before the aforementioned onlinenew policies learning are learned; performance, e.g.,(2) (1)smooth changesmodel fine in traffic tuning could variability notmight be enough produce toloss mitigate beforepersistent new policies errors in taking are learned; (2)some smooth specific modelactions; fine tuningandcould (3) online learning not be enough to mitigate persistent errors in taking some specific actions; and (3) online ac- tends to forget valuable learning in the long run, thus reducing the model’s curacy [7]. tends to forget valuable learning in the long run, thus reducing the model’s learning accuracy Figure[7]. 2a represents a possible evolution of the traffic variation (the traffic pattern is Figure omitted here for 2a represents simplicity)a and possible evolutionperformance—overprovisioning, the obtained of the traffic variation (the traffic pattern traffic loss, is omitted here for simplicity) and some other QoS metric. The path supportingand the obtained performance—overprovisioning, the flow is established at timetraffic t0 and the loss, and some other QoS metric. The path supporting the flow is established at time t0 and desired QoS is specified, so the RL algorithm needs time to learn the traffic variation (and the desired QoS is specified, so the RL algorithm needs time to learn the traffic variation the(and traffic pattern); meanwhile (until ta in Figure 2a), poor performance, including traffic the traffic pattern); meanwhile (until ta in Figure 2a), poor performance, including loss, traffic loss,expected. can be Once a good can be expected. Once model a good is obtained, model it is expected is obtained, that an it is expected thatRLanalgorithm RL canalgorithm provide can the target provideperformance. However,However, the target performance. a steep change a steepin the variation change of the traffic in the variation (times of thet1 traffic to t2) can (timesimpact t1 to tthe performance 2 ) can until the new impact the performance variation until the new is learned. variation is Nonetheless, learned. Nonetheless, it might happen that the performance does it might happen that the performance does not converge to the desire level evennot converge to the desire level after even after learning the learning the new traffic variation.new traffic variation. (a) Traffic Variation Traffic Variation (b) Detection Detection Traffic variation Traffic variation t0 ta t1 t2 time t0 ta' tb’ t1 t2 tc' time Path Traffic Path Off-line-trained Traffic Off-line-trained Set-up Change Set-up model Change model Pre-trained Traffic Loss Overprovisioning Traffic Loss On-line model Learning Phase I Phase II Phase III Phase II Phase III time time Overprovisioning Self-tuned threshold On-line Learning time time QoS QoS time FigureFigure 2. Operation lifecycle. 2. Operation (a) (a) lifecycle. Online learning Online learningRL RLoperation. operation. (b) Offlinetraining (b) Offline training with with online online finefine tuning tuning RL operation. RL operation. It It seemsclear seems clearthat that the the above abovebehavior behavioris isunacceptable unacceptable for for network operators, network as it as it operators, would provide poor performance and might incur penalties due to SLA breaches. Specifi- would provide poor performance and might incur penalties due to SLA breaches. Specif- cally, it seems of paramount importance to start the operation with already trained models. ically, it seems of paramount importance to start the operation with already trained mod- To that end, an initial model can be trained offline using a network simulator in a sandbox els.domain. To thatOnce end,inan initial model operation, can will the model be trained offline be improved by using a network the online learner. simulator However, in a sandbox there are domain. Once in operation, traffic characteristics, thepattern, e.g., traffic model that willarebeobserved improved bya the after longonline period learner. of However, time, e.g., several days. Therefore, some alternatives are needed to operate the flow duringa long there are traffic characteristics, e.g., traffic pattern, that are observed after period of time, that initial e.g., several days. Therefore, some alternatives are needed to operate the time. Our solutions flow during go beyond that initial time. training offline and propose implementing offline–online learning cycles to deal Our solutions go beyond with large changes training in traffic offline andflow, i.e., toimplementing propose provide guaranteed per- offline–online formance during the whole lifetime of the traffic flow (Figure 2b). Specifically, (i) a policy- learning cycles to deal with large changes in traffic flow, i.e., to provide guaranteed per- based management implemented as a self-tuned threshold-based algorithm is in charge of formance during the whole lifetime of the traffic flow (Figure 2b). Specifically, (i) a policy- managing the flow capacity during the time immediately after the path is set up (Phase I: based timemanagement implemented interval [t0 , ta’], as a self-tuned where ta’-t0 should threshold-based be short, e.g., algorithm 1 h). That algorithm tunesisa in charge of thresh- Sensors 2021, 21, 8306 6 of 24 Sensors 2021, 21, 8306 managing the flow capacity during the time immediately after the path is set up (Phase I: 6 of 24 time interval [t0, ta’], where ta’-t0 should be short, e.g., 1 h). That algorithm tunes a thresh- old for the flow capacity and accurately determines the traffic variation. This approach enables dynamic flow capacity allocation by fixing the right values for the threshold that old for theoverprovisioning. minimize flow capacity andHowever, accurately determines avoiding the traffic traffic loss and variation. guaranteeingThisthat approach the re- enables dynamic flow capacity allocation by fixing the right values for the threshold quired QoS is met is not a straightforward task, as it depends on the variance in the traffic that minimize overprovisioning. However, avoiding traffic loss and flow—defined as the difference between maximum and minimum amount of traffic dur-guaranteeing that the required QoS is met is not a straightforward task, as it depends on the variance ing some period. Therefore, during this period, the threshold is set conservatively to avoid in the traffic flow—defined as the difference between maximum and minimum amount of traffic during underprovisioning (i.e., traffic exceeds capacity, and some traffic is loss) at the expense of some period. Therefore, during this period, the threshold is set conservatively to avoid un- large overprovisioning. (ii) Once the variation of the traffic has been determined, a pre- derprovisioning (i.e., traffic exceeds capacity, and some traffic is loss) at the expense of large trained generic model can be used for flow operation (Phase II: time interval [ta’, tb’]). The overprovisioning. (ii) Once the variation of the traffic has been determined, a pre-trained model is general as it has been pre-trained assuming a given traffic pattern, e.g., sinusoidal generic model can be used for flow operation (Phase II: time interval [ta’, tb’]). The model is with daily periodicity, but supporting the measured traffic variation. Once in operation, general as it has been pre-trained assuming a given traffic pattern, e.g., sinusoidal with the pre-trained model starts to fine tune with the observed samples. (iii) Once enough daily periodicity, but supporting the measured traffic variation. Once in operation, the measurements are available to determine the characteristics of the flow, including the traf- pre-trained model starts to fine tune with the observed samples. (iii) Once enough measure- fic pattern, a specific model can be trained in a sandbox domain by using a simulator set ments are available to determine the characteristics of the flow, including the traffic pattern, to operate at time tb’ (Phase III). That model should improve the performance or be easier a specific model can be trained in a sandbox domain by using a simulator set to operate to operate than the pre-trained one. (iv) Assuming that the traffic pattern does not change, at time tb’ (Phase III). That model should improve the performance or be easier to operate any thanchange in the traffic the pre-trained one. variation that cannot (iv) Assuming that thebe absorbed traffic patternbydoes the current model not change, anycan trig- change ger returning to Phase II for more intensive parameter tuning for the new traffic in the traffic variation that cannot be absorbed by the current model can trigger returning variation, while to Phasea new specific II for model is parameter more intensive trained andtuning is set to foroperate the newattraffic time tc’ (Phase II—Phase variation, while a new III cycle). specific model is trained and is set to operate at time tc’ (Phase II—Phase III cycle). 2.2. Proposed Proposed Architecture This work extends extends the the basic basic RL-based RL-based flowflow capacity capacity operation operation (Figure (Figure 1) 1) and and pro- pro- poses a scheme based on (Figure 3): (i) analyzing the traffic to obtain meaningful traffic characteristics; characteristics; (ii) (ii)making makingdecisions decisionsregarding regardingthe theallocated allocatedcapacity capacity whenwhennonomodel model is in is operation (Phase in operation I); (iii) (Phase selecting I); (iii) pre-trained selecting models pre-trained that fit models thatwith the observed fit with traffictraffic the observed char- acteristics; (iv) training characteristics; new new (iv) training models in a sandbox models in a sandboxdomain (offline domain learning), (offline where learning), real traf- where real fic measurements traffic measurements areareusedusedto to generate generatetraffic trafficinina asimulation simulationenvironment environment and and the the QoS estimated; a replica of the RL algorithm in operation is used here for can be realistically estimated; training new models; training models; and and (v) once accurate accurate models models are obtained, obtained, they are used for flow operation and operation andwill willbebeprogressively progressivelyfine-tuned fine-tunedonline. online. AsAs in Figure in Figure a Flow 1, a1,Flow Manager Manager col- collects lects monitoring monitoring datadata fromfromthethe forwarding forwarding plane plane andand enforces enforces flow flow capacity. capacity. From SDN Controller Train new Model QoS (params) Analyzer Set () Offline Learning Decision Making (Sandbox Domain) New Model RL-based Operation Traffic Reward RL Framework RL Framework Analyzer Analyzer (Learner, Agent, Environment) (Learner, Agent, Environment) Reward Traffic, Required Capacity Traffic, Required Capacity Capacity Required Capacity Capacity (Phase II and III) Flow Simulator (Phase I) Flow Manager Traffic, QoS Capacity Metering Estimation Traffic DB Monitoring Capacity Traffic Flow Generation Simulation Flow Figure 3. Extended architecture for flow capacity autonomous operation with offline offline learning. learning. The models include some parameters that need to be tuned as a function of the traffic, to provide the desired performance while meeting the required QoS. Such parameter tuning can be carried out during offline learning, as well as during online operation to deal with small traffic changes. Based on the analysis of the traffic and the reward, the Analyzer block decides when to tune parameters and when to update the model with an offline learned one (labeled Set() in Figure 3) to meet the given QoS. Note that both parameter Sensors 2021, 21, 8306 7 of 24 tuning and the offline–online cycle can be completed several times during the operation to improve the learned models, which will also enable adaptability to changes. The next section details the RL approaches used to solve the autonomous flow capacity management problem, CRUX. 3. CRUX Problem Definition and RL Methodology This section formally defines the CRUX problem, and introduces the main parameters and variables used hereafter. Next, it introduces the methodology to solve the problem using RL, and finally defines the different RL approaches under study. The used notation is summarized in Table 1, where parameters and variables are defined. Table 1. Notation. Capacity and QoS Params for the Flow zmax Maximum capacity (Gb/s) dmax Target maximum delay (s) l* Optimal load (unleashing dmax ) ∈ [0, 100]% qa QoS Assurance (%) Traffic and Capacity xmax (t) Maximum traffic at time t (Gb/s) xvar (t) Traffic variation at time t (Gb/s) z(t) Capacity allocated at time t (Gb/s) o(t) Capacity slack/surplus at time t (Gb/s) y(t) Overprovisioning margin (Gb/s) ρ Traffic variance multiplier b Granularity of capacity allocation (Gb/s) w(t) Traffic loss margin (Gb/s) Autonomous Capacity Allocation a(t) Action time t (b/s) na Number of discrete actions s(t) State at time t ns Number of discrete states r(t) Reward at time t k Threshold-based scaling factor βi Reward function coefficients (≥0) 3.1. Problem Definition and Basic Modeling Let us consider that the autonomous flow capacity management problem is solved periodically, when a new set of measurements, statistics, and parameters for the traffic flow x are collected and computed. Along this section and the following, we adopt the informational representation of time, where t represents a point in time that refers to the time interval [t − 1, t) [22]. Specifically, x(t) represents the traffic measurements collected in [t - 1, t), and statistics, such as the maximum (xmax (t)) and variation (xvar (t)), summarize traffic dynamicity during that time interval. Additionally, decisions made at time t, e.g., the capacity to be allocated (z(t)), consider data that arrived up to time t. The main objective of the CRUX problem is to find the optimal (minimum) capacity z(t) at time t that satisfies a desired QoS. In this work, we assume that that the QoS is defined by a desirable maximum end-to-end delay dmax ; this also entails that packet loss is not tolerated during flow operation. Without loss of generality, we assume that the delay can be modeled as a function of the traffic volume, the allocated capacity for the flow, the load (ratio traffic/capacity), and other components such as the transmission delay (load–delay models). Such load–delay models can be obtained during the commissioning testing phase using, e.g., active monitoring techniques [20]. Once the model is available, a target load l* unleashing the target maximum delay dmax can be selected. Figure 4 illustrates an example of a load–delay model, where other components such as the transmission delay (load–dela models can be obtained during the commissioning testing ph Sensors 2021, 21, 8306 toring techniques [20]. Once the model is available, a8 oftarget 24 l maximum delay dmax can be selected. Figure 4 illustrates a model, where dmax has been selected to a value where queue dmax has been selected to a value where queueing delay becomes the predominant delay dominant component, e.g.,delay component, e.g., for l* = 80%. for l* = 80%. delay dmax 0 load (l) l* 1 Figure 4. Delay model example. Figure 4. Delay model example. A policy (threshold)-based approach can be used to make decisions from the currently available monitoring data, as defined in Equation (1), where k is a constant factor that A policy (threshold)-based approach can be used to m is related to the traffic dynamicity and variability; k needs to be tuned to guarantee the required QoS. However, finding the proper value of k is not a straightforward task: if the rently available monitoring data, as defined in Equation (1), value of k is high, QoS is ensured at the cost of high overprovisioning, whereas if the value that is related to the traffic dynamicity and variability; k nee of k is low, QoS requirements might be not met. In addition, decisions are reactive, so altogether, sub-optimal solutions are usually obtained. the required QoS. However, finding the proper value of k is n z(t) = k · xmax (t)/l ∗ (1) the value of k is high, QoS is ensured at the cost of high over The optimal capacity allocation to the CRUX problem requires knowledge of the value of k is low, QoS requirements might be not met. In add expected traffic to allocate the capacity of the flow at time t − 1 to the value that fits the so altogether, expected maximum load sub-optimal for the period [tsolutions aredecision − 1, t) (proactive usually obtained. making): z∗ (t − 1) = xmax (t)/l ∗ (2) 𝑧(𝑡) = 𝑘 · 𝑥𝑚𝑎𝑥 (𝑡)⁄𝑙 ∗ In the case that the capacity allocation is not optimal, some capacity slack/surplus (o) The optimal capacity allocation to the CRUX problem re will exist, which can be formally computed at time (t) as follows: pected traffic to allocate o (t) = z(t the − 1) −capacity xmax (t)/l ∗ of the flow at time (3) t− expected maximum Figure 5 sketches an exampleload for flow of a traffic thex(t)period for which[t - 1,capacity some t) (proactive allocation dec z(t) is required. In the figure, the optimal capacity z*(t) that should be allocated at time ∗ of the values of o(t). In ∗ 𝑧 (𝑡 − 1) = 𝑥 ti −1 is shown. The different colors provide a visual representation particular, two different sub-optimal capacity allocations can be distinguished 𝑚𝑎𝑥 (𝑡)⁄𝑙 (see labels in Figure 5a): (i) if z(t) > z*(t) (i.e., o(t) > 0), QoS requirements are met at the expense of an In the case that the capacity allocation is not optimal, som excess of overprovisioning; (ii) if z(t) < z*(t) (i.e., o(t) < 0), QoS requirements are violated. will exist, which can be formally computed at time (t) as follo 𝑜(𝑡) = 𝑧(𝑡 − 1) − 𝑥𝑚𝑎𝑥 (𝑡)⁄𝑙 ∗ Figure 5 sketches an example of a traffic flow x(t) for wh z(t) is required. In the figure, the optimal capacity z*(t) that sh is shown. The different colors provide a visual representation ticular, two different sub-optimal capacity allocations can be SensorsSensors 2021, 2021, 21, 21, 8306 8306 9 of 249 of 24 Sensors 2021, 21, 8306 9 of 24 (a) overprovision z*(t0) (b) z*(t2) (a) o(t) >0 z*(t ) 1(b) z*(t traffic or capacity 2) overprovision z*(t0) xmax(t3) z*(ti-1) o(t) > 0 xmax(t1) z*(t ) traffic or capacity z*(t 1 xmax(t2) xmax(t3) i-1) overprovision xmax(t1) xmax(ti) w(ti) o(t) < 0 (delay) xmax(t2) overprovision xmax(ti) w(ti) o(t) < 0 (delay) underprovision x (ti) underprovision x (ti) o(t) < -w(t) (loss) o(t) < -w(t) (loss) ti-1 ti-1 timetime ti t tt0 tt11 t2 t2 t3 t3 i 0 Figure 5. Capacity Figure Figure5.5. allocation Capacity allocation Capacity definition definition allocation (a) definition and (a)(a) and evolution evolution and (b). evolution(b). (b). The The The width width width w(t) w(t) w(t) ofofthe of the the high highdelay highdelayarea delay is formally area area is defined is formally formally as a as defined defined function of theofmaximum a function as a function the of maxi- the maxi- traffic mummum in trafficEquation traffic in in (4). Therefore, Equation(4). Equation traffic (4).Therefore, loss appears Therefore, traffic traffic loss if o(t) lossappears appears ≤ − if ifw(t). o(t) ≤ −w(t). o(t) ≤ −w(t). 1 −1 l−∗ 𝑙∗ ∗ 𝑤(𝑡) t) = = w(𝑤(𝑡) 𝑥𝑥𝑚𝑎𝑥 x=max ∗(𝑡)∗∗∗1 ∗− 𝑙 (t)(𝑡) (4) (4) (4) 𝑚𝑎𝑥 l 𝑙 𝑙∗ It is is worth noting that the quality ofof a solution taken at time t − 1t can1only be evalu- It isIt worth worth noting noting thatthatthethe qualityof quality a solution a solution taken taken atattime time t −− 1 cancan only only bebeevalu- ated at evaluated time at t, timewhich t, motivates which the motivates use the of use RLof to RL learn to the learn optimal the optimal policy that policy allocates that the allocates atedminimum at time t,value which of motivates z(t) to meet the the use QoS ofrequirements. RL to learn the The optimal details ofpolicy the that allocates RL-based meth- the the minimum value of z(t) to meet the QoS requirements. The details of the RL-based minimum odologyvalue of z(t) toinmeet are presented next the QoS requirements. The details of the RL-based meth- subsections. methodology are presented in next subsections. odology are presented in next subsections. 3.2. Generic 3.2. Generic RL-Based RL-BasedMethodology Methodology 3.2. Generic RL-Based Figure Figure Methodology 66 illustrates illustrates the RL the RL workflow, workflow,where wherethe themain mainthree threeelements elementsinvolved involvedare are represented, represented, namely:(i) namely: Figure 6 illustrates (i)the thelearner the learner RL inincharge workflow,chargewhere oflearning of learning theoptimal the the main optimal policy; three policy; elements (ii)the (ii) the agent are agent involved in in charge charge of of taking taking actions actions to to adjust adjust the the capacity capacity allocated allocated to to the the represented, namely: (i) the learner in charge of learning the optimal policy; (ii) the agent flow; flow; (iii) (iii) the the environment environment adaptation module in in charge of ofimplementing adaptation in charge of taking moduleactions charge to adjust the capacityand implementing and evaluating evaluating allocated the actions theflow; to the actions taken; (iii)taken; and and (iv) (iv) the environment the the flow flowmanager, manager,which whichenforces enforcesthe thecapacity capacityand andcollects collectstraffic trafficmeasurements. measurements.Three Three adaptation module in charge of implementing and evaluating the actions taken; and (iv) time timeperiods periodsare arespecified, specified,fromfromt0t0to tott22;; let let us usassume assumethat thatsome someinitial initialpolicy policymodelmodelhashas thebeen flow been set manager, set ininthe which theagent agentbefore enforces beforeoperation the capacity operationstartsstartsatattime and collects timet0t0,, when whenthe traffic theagent measurements. agentapplies appliesthethefirst Three first time periods action action a(t00).).are specified, from t0 to t2; let us assume that some initial policy model has a(t been set in the agent before operation starts at time t0, when the agent applies the first action a(t 0). Learner Agent Environment Flow Mgr model Learner Agent Environment Flow Mgr t0 a(t0) Allocate z(t0) model Mon. data Mon. data t0 a(t0) Allocate … z(t0) Mon. data s(t1), r(t1) Mon. data t1 Allocate Mon. z(t1) data a(t1) a(t1) Mon.…data Mon.data Mon. data s(t1), r(t1) … t1 Allocate Mon. dataz(t1) a(t1) s(t a(t2),1)r(t2) Mon. data t2 Mon. data Allocate a(t2) a(t2) … z(t2) Learn Mon.data Mon. data model s(t2), r(t2) Mon. data t2 … a(t2) a(t2) Allocate z(t2) FigureLearn Figure6.6.General GeneralRLRLworkflow. workflow. Mon. data model Mon. data … complexity, action a(t) is defined in Equa- For the sake of simplification and to reduce tion (5) as the differential capacity with regard to the current one. Actions are processed Figure 6. General by the RL workflow. environment, which computes the new capacity z(t) to be allocated. For the sake of simplification and to reduce complexity, action a(t) is defined in Equa- tion (5) as the differential capacity with regard to the current one. Actions are processed by the environment, which computes the new capacity z(t) to be allocated. 𝑎(𝑡) = 𝑧(𝑡) − 𝑧(𝑡 − 1) Sensors 2021, 21, 8306 10 of 24 The flow manager periodically sends traffic monitoring data to the which processes them at the end of every time interval to compute state s r(ti).ForUpon receiving the sake the state, of simplification and the agentcomplexity, to reduce finds theaction action a(t)a(t ) to be intaken w is idefined policy.(5) Equation Inasaddition, thecapacity the differential learningwithprocess regard to uses state, the current reward, one. and Actions are pro-action t cessed by the environment, which computes the new capacity z(t) to be allocated. model, which is updated in the next time interval. The state function s(t terms of o(t) normalized by a(t)a=parameter z(t) − z(t − 1)y(t) (Equation (6)), which(5)is conv using parameter ρ to absorb the traffic variation observed in the flow (Equ The flow manager periodically sends traffic monitoring data to the environment, which processes them at the end of every time interval state s(ti ) and reward 𝑠(𝑡) to=compute 𝑜(𝑡)/𝑦(𝑡) r(ti ). Upon receiving the state, the agent finds the action a(ti ) to be taken with the current policy. In addition, the learning process uses state, reward, and action to improve the model, which is updated in the next time interval. The𝑦(𝑡state ) = function (𝑡)is defined in terms 𝜌 ∙ 𝑥𝑣𝑎𝑟s(t) of o(t) normalized by a parameter y(t) (Equation (6)), which is conveniently set up using parameter As ρfor the reward to absorb r(t), the traffic the objective variation observed inisthe toflow minimize (Equationoverprovisioning, (7)). w ing high delay. To ensure that, the higher delay must be obtained by pr s(t) = o (t)/y(t) (6) overprovisioning. To that end, we have defined a piece-wise function with regions in the range of o(t),y(representing t) = ρ · xvar (t) the sub-optimal cases. Such (7) divi individual As for the modes reward r(t),of the operation that objective is correspond to minimize to an adequate overprovisioning, withoutreward pro- in viding high delay. To ensure that, the higher delay must be obtained reward function r(t) is formally expressed in Equation (8) and illustrated in by producing some overprovisioning. To that end, we have defined a piece-wise function with four different first and regions second in the range of components of r(t) o(t), representing the penalize sub-optimaltraffic cases. loss Such and high division delay, res allows components for individual modesare linear functions of operation of o(t), where that correspond coefficients to an adequate reward β andcase. in1 each β2 can be t The reward function r(t) is formally expressed in Equation ize traffic loss and high delay. The third component gives the maximum(8) and illustrated in Figure 7. re The first and second components of r(t) penalize traffic loss and high delay, respectively. slightly Both shifted components areto the functions linear positiveofvalues of coefficients o(t), where o(t) to reduce β1 andthe riskbeof β2 can QoS viola tuned ment to is concave penalize quadratic traffic loss and high delay.with The regard to the gives third component relation o(t)/y(t),reward, the maximum with the m which is slightly shifted to the positive values of o(t) to reduce the weighted by coefficient β3. Finally, overprovisioning above y(t) is linearly risk of QoS violation. This segment is concave quadratic with regard to the relation o(t)/y(t), with the maximum coefficient value weightedβby 4. coefficient β3 . Finally, overprovisioning above y(t) is linearly penalized by coefficient β4 . 𝛽1 · (𝑜(𝑡) − 𝑤(𝑡)) + 𝛽2 · 𝑤(𝑡), 𝑜(𝑡) < −𝑤(𝑡) β 1 ·(o (t) − w𝛽(2t))· 𝑜(𝑡), −𝑤(𝑡) ≤ 𝑜(𝑡) < 0    + β 2 · w ( t ), o ( t ) < − w ( t ) ·o (t),=   β 2𝑟(𝑡) 𝑜(𝑡) 𝑜(𝑡) − w ( t ) ≤ o ( t ) < 0  r (t) = o (t) 𝛽3o·(t(1 ) − ) ·0 ≤ o(t,) < y(t) 0 ≤ 𝑜(𝑡) < (8) 𝑦(𝑡)   β 3 · 1 − y ( t ) · y ( t ) , 𝑦(𝑡) 𝑦(𝑡)  − β 4 ·(o (t) { −−𝛽 y(t))·, (𝑜(𝑡) − 𝑦(𝑡)), o (t) ≥ y(t)  4 𝑜(𝑡) ≥ 𝑦(𝑡) 20 underprovision 0 overprovision -20 Reward (r) -40 -60 Traffic loss High delay -80 -100 -40 -20-w 00 y20 40 Capacity Slack / Surplus (o) Figure Figure7. Reward function 7. Reward vs. capacity function vs.slack/surplus. capacity slack/surplus. 3.3. Specific Adaption of RL Approaches As introduced in Section 1, three RL methods are considered to solve th lem, namely: (i) Q-learning, (ii) D3QN, and (iii) TD3. For each method, the main adaptation of the generic problem definiti Sensors 2021, 21, 8306 11 of 24 3.3. Specific Adaption of RL Approaches As introduced in Section 1, three RL methods are considered to solve the CRUX problem, namely: (i) Q-learning, (ii) D3QN, and (iii) TD3. For each method, the main adaptation of the generic problem definition is to discretize state and action spaces. Q-learning requires discretizing the continuous state function s(t) in Equation (6) into a number of discrete states (ns ). Such discrete state function s’(t) can be Sensors 2021, 21, 8306 formally expressed as Equation (9). Discrete states 0 and ns − 1 indicate underprovisioning 11 of 24 and overprovisioning above margin y(t), respectively, whereas the rest states ns − 2 are used to evenly discretize the overprovisioning below margin y(t).   0, 0, 𝑠(𝑡) s(t)<<0 0 s0(t=) =⌈(𝑛 (−ns2)−· 2𝑠(𝑡)⌉, 𝑠′(𝑡) )·s(t), 𝑠(𝑡) s(t)∈∈[0,1] [0, 1] (9) (9) 𝑛 − ns1,− 1, 𝑠(𝑡) s(t)>>1 1  In addition, Q-learning In addition, and and Q-learning all DQN variants all DQN require variants a discrete require space a discrete of naofactions. space na actions. Let us define a’(t) as the set of discrete actions, where a discrete action is defined Let us define a’(t) as the set of discrete actions, where a discrete action is defined by an by an integer number of units of capacity b to update (add or subtract) the current capacity. integer number of units of capacity b to update (add or subtract) the current capacity. The The discrete set of discrete setactions thatthat of actions depend on both depend na ∊n2a *∈ 2 * ℕ − 1 (natural on both − 1 (natural odd number) odd number) and b can and be b can be formally formally defineddefined as: as: 𝑛 − 1 𝑛 − 1  𝑎 (𝑡) na − 1 na − 1  a0 (t∈) ∈𝑏 · 𝑖, b·𝑖i,∈i ∈− −2 , , 2 (10) (10) 2 2 Table 2 summarizes the main characteristics and parameters to be configured for Table 2 summarizes the main characteristics and parameters to be configured for each method. each method. Table 2. Summary of RL approaches. Table 2. Summary of RL approaches. Approach State Space Action Space Parameters Approach Discrete, State Q-learning Space Equation Action (10) (9) Discrete. Equation Space ns, naParameters ,b Q- na, b, DNN config, Replay D3QN Continuous, Equation Discrete, (6)(9)Discrete. Equation Equation Discrete. (10) Equation (10) n ,n ,b learning buffer s a Continuous, Equation naDNN Actor/critic , b, DNN config, config TD3D3QN Continuous, Continuous, Equation Equation (6) (6) Discrete. Equation (10) (10) Replay Replay buffer buffer Actor/critic DNN config TD3 Continuous, Equation (6) Continuous, Equation (10) Replay buffer 4. Cycles for Robust RL This section is devoted to the details of the operation lifecycle presented in Figure 2b. 4. Cycles for Robust RL Let us assume that when a new path for a flow is set up, an instance of every element in This section the architecture is devoted in Figure to the details 3 is instantiated. Theofcharacteristics the operation lifecycle presented of the instances in Figure depend on 2b. Let us assume that when a new path for a flow is set up, an instance of every the flow requirements, e.g., pre-trained generic models loaded in the Offline Learning block element in the are those that were trained with similar QoS requirements (dmax and qa) to those of the the architecture in Figure 3 is instantiated. The characteristics of the instances depend on flowflow. current requirements, When thee.g., pre-trained operation starts,generic models no online loadedexist RL models in the Offline and Learning block the maximum ca- are those that were trained with similar QoS requirements (dmax and qa ) to those of the current pacity for the flow zmax is allocated. All the algorithms presented next run in the Analyzer flow. When the operation starts, no online RL models exist and the maximum capacity for block (see Figure 3), which makes decisions and orchestrates the rest of the blocks based the flow zmax is allocated. All the algorithms presented next run in the Analyzer block (see on some analysis results. Figure 3), which makes decisions and orchestrates the rest of the blocks based on some Algorithm 1 shows the Analyzer initialization that receives the pointers to external analysis results. modules that interact with the Analyzer, i.e., offline learner, online RL, and flow manager, and Algorithm 1 shows the Analyzer initialization that receives the pointers to external stores them (line 1 in Algorithm 1), and initializes the main variables used by the rest of modules that interact with the Analyzer, i.e., offline learner, online RL, and flow manager, and the procedures. In particular, DB contains the needed traffic-related data for the analysis stores them (line 1 in Algorithm 1), and initializes the main variables used by the rest of carried out at every phase (line 2), params is a vector with the parameters that characterize the procedures. In particular, DB contains the needed traffic-related data for the analysis flow’s requirements and some configuration and that are used during the different phases carried out at every phase (line 2), params is a vector with the parameters that characterize (line 3), and phase records the current phase, which is initialized to Phase I (line 4). flow’s requirements and some configuration and that are used during the different phases (line 3), and phase records the current phase, which is initialized to Phase I (line 4). Algorithm 1. Analyzer Initialization INPUT: offlineLearn, onlineRL, flowMgr OUTPUT: - 1: store(offlineLearn, onlineRL, flowMgr) 2: initialize DB 3: params ← [, // Phase I , // Phase II Sensors 2021, 21, 8306 12 of 24 Algorithm 1. Analyzer Initialization OUTPUT: - INPUT: offlineLearn, onlineRL, flowMgr 1: store(offlineLearn, onlineRL, flowMgr) 2: initialize DB 3: params ← [, // Phase I , // Phase II ] // Phase III 4: phase ← PhaseI Before describing the algorithms for the different phases, let us present a specific procedure for traffic variance analysis. Algorithm 2 is applied to the set of observed traffic- related measurements, stored in DB, with the objective of characterizing and quantifying the fluctuation of the traffic around its observed average. After retrieving data time series contained in DB, the average pattern on the given traffic time series X is computed (lines 1–2 in Algorithm 2). Note that the result of this operation produces time series Xavg , with the smoothed average that better fits X. Without loss of generality, we assume that a combination of regression techniques including polynomial fitting, spline cubic regression, and sum-of-sin regression is applied, returning the best result in terms of accuracy and model complexity [23]. The relative residuals are computed (line 3) and the difference between maximum and minimum of these residuals (var) is considered as the traffic variance measurement (line 4). This value together with the previous var measurements stored in DB (time series Y) are used to compute the derivative drv of var in time, i.e., the first-order difference [24] (lines 5). The last value in drv denotes the current derivative, which is later used for the identification of the traffic variance (line 6). In addition, a variance score is computed as the maximum absolute derivative value normalized by the observed variance (line 7). The score approaches 1 if the traffic fluctuates around its maximum range between two consecutive time measurements. This score will be used later for generic model tuning purposes. The computed variance results are eventually returned (line 8). Algorithm 2. varianceAnalysis(). INPUT: DB OUTPUT: V 1: ← 2: Xavg ← computeAveragePattern(X) 3: Xres ← (X—Xavg ) X−1 4: var ← max(Xres )—min(Xres ) 5: Y.append(var) 6: drv ← computeDerivative(Y) 7: score ← max(|drv|)/var 8: return Algorithm 3 specifies the main procedure running in the Analyzer block and it is called periodically every time t. First, new traffic monitoring data are gathered from the flow manager (line 1 in Algorithm 3). Then, the specific procedure for each phase is called with the current time and the collected data, and the phase changes only when the called procedure returns True (lines 2–10); DB is initialized every time phase changes (line 11). Algorithm 4 defines the operation of the Analyzer block during Phase I. Recall that dur- ing this phase, flow capacity allocation is managed following a threshold-based procedure, defined by Equation (1). First of all, DB is updated with the new monitoring data and the variance analysis described in Algorithm 2 is executed, storing the result in variable V (lines 1–2 in Algorithm 4). Then, the absolute value of the current derivative is compared with a small epsilon value (param eps) to decide whether enough traffic data have been already analyzed to estimate variance with high accuracy (line 3). If so, a generic pre-trained model f0 for the computed variance is retrieved from the offline learner and factor ρ0 is scaled with the ratio of scores between the generic model and the observed traffic (lines 4–5); the scaled Sensors 2021, 21, 8306 13 of 24 factor increases (decreases) if the computed score is higher (lower) than the score of the generic model. The rationale behind such factor correction is to achieve a more robust and conservative operation of the generic model under the actual traffic variance behavior. Then, the online RL module is updated with new model f0 and scaled factor ρ1 and Phase I ends (lines 6–7). Algorithm 3. Main Analyzer Procedure INPUT: t OUTPUT: - 1: x(t)←flowMgr.getMonitoringData(t) 2: if phase = PhaseI then 3: changePhase ← thresholdBased(t, x(t)) 4: if changePhase then phase ← PhaseII 5: else if phase = PhaseII then 6: changePhase ← modelSelectionAndTuning(t, x(t)) 7: if changePhase then phase ← PhaseIII 8: else // phase = PhaseIII 9: changePhase ← specificModel(t, x(t)) 10: if changePhase then phase ← PhaseII 11: if changePhase then initialize DB Algorithm 4. thresholdBased() (Phase I) INPUT: t, x(t) OUTPUT: changePhase 1: DB.traffic.append(x(t)) 2: V ←varianceAnalysis(DB) 3: if |V.curdrv| < eps then 4: ← offlineLearn.getGenericModel(V.var) 5: ρ1 ←ρ0 ·(sc0 / V.score) 6: onlineRL.setModel(f0 , ρ1 ) 7: return True 8: DB.var.append(V.var) 9: k(t)←max(1, V.var / (1−l*)) 10: if k(t)>k then k←k(t) else k←k − (k(t) − k)/2 11: flowMgr.setupCapacity(max(k · max(x)/l*, zmax )) 12: return False In case the current derivative is still high, the threshold-based capacity allocation procedure continues. Here, factor k in Equation (1) is adapted from its input value as soon as more traffic data are available and traffic variance is better estimated. With the estimated variance and target load l*, factor k(t) is computed. Then, k is updated in two different ways: (i) reducing by half between k and k(t) if k is larger than needed; or (ii) replaced by k(t) if is k is lower than needed (lines 8–10). The flow manager is requested to modify the flow capacity to the computed z(t), which is bounded by the maximum capacity zmax (line 11) and Phase I continues (line 12). Algorithm 5 details the procedure during Phase II. Traffic data collected from the flow manager are sent to the offline learner block for updating a historical traffic database used to train RL models offline in the sandbox (line 1 in Algorithm 5). The offline model training procedure runs in parallel in the sandbox domain. When enough traffic measurements are collected and processed and an accurate and robust offline-trained RL model is available, it is sent to the online RL block to be used for flow capacity operation; this ends Phase II (lines 2–5). Otherwise, Phase II continues, aiming to identify whether the RL model currently in operation needs some parameter tuning (in addition to model updates that the RL algorithm performs during operation). In particular, this procedure aims to supervise the degree of QoS assurance (as compared with the target value, qa) obtained by the current model and modifying factor ρ when needed to achieve the target performance. Note that low overprovisioning is the secondary objective and therefore, QoS assurance analysis Sensors 2021, 21, 8306 14 of 24 requires computing whether the current capacity violated maximum delay (o(t) < 0) or not (o(t) ≥ 0). The result is stored in DB (lines 6–10). Next, two different one-sample proportion binomial hypothesis tests [25] are conducted to detect whether the observed degree of QoS assurance is significantly below (test 1) or above (test 2) the target value qa (lines 11–13). In the case that some of the hypotheses can be confirmed (on the contrary, it is assumed that QoS assurance is in the target), ρ needs to be tuned. If hypothesis test 1 is confirmed, some extra capacity allocation is needed, which is achieved by increasing ρ with a given step size ∆ρ. On the contrary, if hypothesis test 2 is confirmed, allocated capacity can be reduced, so ρ is decreased by the same step size. Algorithm 5. ModelSelectionAndTuning() (Phase II) INPUT: t, x(t) OUTPUT: changePhase 1: offlrn.updateTrafficDB(x(t)) 2: if offlineLearn.newModelAvailable() then 3: ← offlineLearn.getModel() 4: onlineRL.setModel(f, ρ) 5: return True 6: xmax (t) = max(x) 7: z(t − 1)← flowMgr.getCurrentCapacity(t) 8: o(t)←computeSlackSurplus(xmax (t), z(t − 1))//Equation (3) 9: if o(t)<0 then DB.QA.append(0) 10: else DB.QA.append(1) 11: pobs ←avg(DB.QA) 12: pval_l←BinomialTest1(“pobs < qa”) 13: pval_g←BinomialTest2(“pobs > qa”) 14: if min(pval_g, pval_l) > cfl then 15: DB.QA←∅ 16: if pval_l < cfl then onlineRL.tuneParam(‘ρ’, ∆ρ) 17: else onlineRL.tuneParam(‘ρ’, −∆ρ) 18: return False Finally, Algorithm 6 describes the procedure running in the Analyzer during Phase III. This algorithm analyzes the last m traffic measurements and the reward obtained by the online RL (lines 1–6 in Algorithm 6). The objective of this analysis is to check whether both the current traffic and reward follow the expected behavior (line 7). Let us assume that an extended estimation of the working variance with range [var_l, var_h] is found during the offline training phase—with a minimum and maximum variance that the RL model can support without losing either robustness or desired performance. Bear in mind that operating a traffic flow with more variance than what is supported by the model can lead to poor QoS assurance and even traffic loss. On the contrary, a traffic flow with less variance can produce large overprovisioning, which the online RL can hardly decrease with its fine adaption configuration. In fact, online RL continuously adapts the model to smooth traffic changes with controlled reward fluctuations, so that a minimum reward (rw_l) can be considered as the reasonable limit of a normal RL operation. Therefore, Phase II is triggered back when traffic variance leaves the working range of the RL model or the observed reward goes below that limit; otherwise, Phase III continues. Algorithm 6. specificModel() (Phase III) INPUT: t, x(t) OUTPUT: changePhase 1: rw(t) ← onlineRL.getReward(t) 2: DB.traffic.append(x(t)) 3: DB.reward.append(r(t)) 4: if |DB.traffic| > m then DB.traffic.pop(0) 5: if |DB.reward| > m then DB.reward.pop(0) 6: V ←varianceAnalysis(DB) 7: return V.var NOT IN [var_l, var_h] OR rw(t) < rw_l Sensors 2021, 21, 8306 15 of 24 5. Illustrative Results For the ongoing evaluation, a Python-based simulator reproducing the modules described in Figure 3 was implemented. Realistic traffic flow behavior was accurately emulated using a simulator based on CURSA-SQ [26]. CURSA-SQ combines statistically based traffic flow generation and continuous G/G/1/k queue model based on the logistic function; multiple queuing systems can be numerically analyzed and related statistics, such as traffic magnitude, queuing delay, and packet loss, be computed. CURSA-SQ was validated with the experimental measurements in [20] and was used to successfully reproduce realistic packet scenarios [11,27]. In the context of this work, CURSA-SQ is used as: (i) a lightweight network simulator to emulate a flow manager and the forwarding plane; and (ii) a flow simulator running in the ML sandbox domain for offline RL training purposes. It is worth highlighting that both CURSA-SQ instances have been independently configured and managed in order to reproduce the actual separation between the physical network and the sandbox domain. Traffic was randomly generated according to different traffic configurations. Each traffic configuration is the combination of traffic pattern and variance. Two different daily patterns were considered: a simple sinusoidal pattern for offline training purposes, and a realistic pattern [16] to emulate the real traffic in the forwarding plane. In both cases, traffic fluctuates between 5 Gb/s (valley) and 40 Gb/s (peak) throughout the day. Regarding variance, it is defined as a percentage of the mean, so the magnitude of traffic oscillations changes in time (heteroscedasticity). For the sake of a wider analysis, we considered five different variance values: 1%, 3%, 6%, 12%, and 25%. RL algorithms running in the RL-based operation and offline learning modules have been implemented in Python3 using libraries such as pytorch. A general epsilon decay strategy was implemented in all the RL methods for balancing between exploration and exploitation [7], with decay factor equal to 0.00125. Moreover, a discount factor equal to 0.95 was set up. Q-learning was configured with ns = 14 states and na = 3 actions, as well as capacity allocation granularity b = 1 Gb/s. In the case of D3QN and TD3, every DNN consisted of two hidden layers with 100 neurons each implementing the Rectified Linear Unit activation function. All DNNs were trained by means of the Adam replacement optimizer with learning rate equal to 0.001 and maximum replay buffer equal to 1e6 samples. Finally, the capacity and QoS parameters for the flow under study are maximum capacity zmax = 100 Gb/s, optimal load l* = 80%, and QoS assurance qa = 99%. In the next two subsections, we first focus on comparing the different RL methods for the scenario where the pure online learning RL-based operation is performed (see Figure 2a). Next, we evaluate the offline leaning + online RL-based operation (Figure 2b), including the three proposed phases. 5.1. Online RL-Based Operation Let us first analyze the reliability of the RL operation under real traffic; we focus specifically on the traffic loss. For this study, low (1%) and high (25%) variances were considered. Figure 8 plots the traffic loss as a function of time from the path set-up time. We observe extremely poor performance (high loss) at the beginning of operation, as it was anticipated in Figure 2a. Interestingly, we observe that the simplest Q-learning method provides the fastest convergence time to achieve zero loss, although it needs more than one day to achieve zero loss operation when traffic variance is high. Note that D3QN is the most sensitive to traffic configuration (zero loss operation time increases three times from low to high variance). TD3 is the method with the slowest convergence (around 4 days). 5.1. Online RL-Based Operation provides the fastest convergence time to achieve zero loss, although it ne Let us first analyze the reliability of the RL operation under real traffic; we focus spe- one day cifically to traffic on the achieve zero loss. For thisloss operation study, low (1%) andwhen traffic high (25%) variance variances wereis high. Note consid- the most ered. Figure sensitive to traffic 8 plots the traffic loss asconfiguration a function of time(zero from loss operation the path time set-up time. We increas Sensors 2021, 21, 8306 observe extremely poor performance (high loss) at the beginning from low to high variance). TD3 is the method with the slowest converge of operation, as it 16 was of 24 anticipated in Figure 2a. Interestingly, we observe that the simplest Q-learning method days). provides the fastest convergence time to achieve zero loss, although it needs more than one day to achieve zero loss operation when traffic variance is high. Note that D3QN is 11 ⨯ 10sensitive 11.E+11 the most to traffic configuration (zero loss operation time increases three times from11.E+10 low to 10 high variance). TD3 is the method Q-Learning 1%with the slowestQ-Learning variation convergence (around 4 ⨯ 10 days). D3QN 25% variation D3QN ⨯ 109 11.E+09 TD3 TD3 Average traffic loss [b/s] ⨯ 1011 11.E+11 ⨯ 108 11.E+08 Q-Learning 1% variation Q-Learning ⨯ 1010 11.E+10 ⨯ 107 11.E+07 D3QN 25% variation D3QN ⨯ 109 11.E+09 TD3 TD3 Average traffic loss [b/s] ⨯1 ⨯8 106 1.E+06 11.E+08 10 107 5 ⨯1.E+05 11.E+07 1 ⨯ 10 ⨯ 106 11.E+06 ⨯ 104 11.E+04 ⨯ 105 11.E+05 ⨯4 10 1.E+03 3 ⨯1 11.E+04 10 ⨯1 ⨯3 102 11.E+03 10 1.E+02 ⨯ 102 11.E+02 ⨯ 101 11.E+01 ⨯ 101 11.E+01 Zero Zero 1.E+00 +0 loss 1.E+00 loss +0 0 0 1 1 2 23 4 3 5 4 5 Time from Time from[dd] path set-up path set-up [dd] Figure Figure 8. 8. Achieving Achieving zero zero loss loss operation. operation. Figure 8. Achieving zero loss operation. As all the RL RL methods methodshavehaveachieved achievedzero zeroloss lossoperation, operation, thetherest of the rest results of the analyze results ana- the performance lyze theAs performanceafter all the RL 5 days after of operation. 5 days of operation. methods As have achieved an illustrative As an illustrative zero loss example example of the of the operation, RL-based the rest of t operation, Figure Figure 99shows showsone oneday dayofofreal traffic real x(t)x(t) traffic andandvariance variance from low low from to high, as well to high, as lyze capacity as the the performance z(t) allocated after using 5 days of optimal Q-learning; operation. As an ρ forρeach illustrative variance example is configured. o well as the capacity z(t) allocated using Q-learning; optimal for each variance is config- operation, The ured.optimal Figure capacity The optimal 9 shows allocation capacity (o(t)one allocation 0)day =(o(t) and of real = 0) the and margin traffic the margin forx(t) and variance for overprovisioning overprovisioning are from l y(t) y(t) also well are plotted. alsoas theWecapacity plotted. observe We that observe z(t)theallocated that allocated the allocatedcapacity using capacityisQ-learning; close to the is close to theoptimal optimalone, optimal absorbing ρ absorb- one, for each vari fluctuations with enough margin to meet the target ing fluctuations with enough margin to meet the target QoS. QoS. ured. The optimal capacity allocation (o(t) = 0) and the margin for overpr 100 are also plotted. 100 We observe that the allocated 100 capacity is close to the optim 90 variance = 1% (a) 90 ing fluctuationsvariance with = 6% (b) enough margin to90 meet the variance target QoS. (c) = 25% ρ = 2.8 ρ = 2.9 ρ = 2.0 Traffic and Capacity [Gb/s] 80 80 80 100 70 y(t) 70 100 70 100 z(t) 90 60 variance o(t) = 1% =0 (a)60 90 variance = 6% 60 (b) 90 variance = 25% x(t) 50 ρ = 2.8 50 ρ = 2.9 50 ρ = 2.0 Traffic and Capacity [Gb/s] 80 80 80 40 40 40 70 30 y(t) 30 70 30 70 20 z(t) 20 20 60 60 60 o(t) = 0 10 10 10 50 00 x(t) 06 03 09 12 15 18 21 24 00 5003 06 09 12 15 18 21 24 00 03 06 0950 12 15 18 21 24 Time (t) [hh] Time (t) [hh] Time (t) [hh] 40 40 40 Figure Figure 9. 9. Q-Learning Q-Learning operation. operation. Traffic Traffic and and allocated allocated capacity capacity for for low (a), moderated low (a), moderated (b), (b), and and high high (c) (c) traffic traffic variance. variance. 30 30 30 Let us now analyze the impact of the margin multiplier ρ to achieve the desired 20 20 20 QoS. Figure 10 shows the obtained QoS assurance as a function of ρ. For the sake of 10 a comprehensive 10 study, all traffic configurations for sinusoidal (Figure 10 10a–c) and real 00 03 06 09 12 15 18 10d–f) (Figure 21 24traffic 00 03 have patterns 06 been09 analyzed. 12 15 The18 minimum 21 24 ρ value 00 03 06 set has been 09 to 12 15 Time (t) [hh]1. The ρ values for which the target QoSTimeof(t)99% [hh]is achieved are highlighted with a round Time (t) [hh] marker. We observe in the results that ρ depends not only on the traffic characteristics, but Figure 9. Q-Learning operation. Traffic and allocated capacity for low (a), moderated (b), and high (c) traffic also on the RL method. Let us now analyze the impact of the margin multiplier ρ to achieve the desired QoS. Figure 10 shows the obtained QoS assurance as a function of ρ. For the sake of a compre- hensive study, all traffic configurations for sinusoidal (Figure 10a–c) and real (Figure 10d– f) traffic patterns have been analyzed. The minimum ρ value has been set to 1. The ρ values Sensors 2021, 21, 8306 for which the target QoS of 99% is achieved are highlighted with a round marker. We 17 of 24 observe in the results that ρ depends not only on the traffic characteristics, but also on the RL method. Q-learning D3QN TD3 1 1 1 1.0 1.0 (a) (b) (c) 0.99 0.99 0.99 1.75 2.95 1.45 1.37 [sinusoidal] QoS Assurance 0.98 0.98 0.98 var=1% var = 1% 0.97 0.97 0.97 var=3% var = 3% var=6% var = 6% var=12% var = 12% 0.96 0.96 0.96 var=25% var = 25% 0.95 0.95 0.95 0.95 1 1 1 1 1 2 3 411.0 2 3 411.0 2 3 4 (d) (e) (f) 1.9 0.99 0.99 0.99 1.67 1.85 QoS Assurance 0.98 6.4 0.98 0.98 [real] 0.97 0.97 0.97 0.96 0.96 0.96 0.95 0.95 0.95 1 2 3 4411 2 3 4411 2 3 4 Margin Multiplier (ρ) Figure 10. QoS as a function of ρ models trained with a sinusoidal traffic pattern (a–c) and real real traffic traffic (d–f). (d–f). Interestingly, Q-learning Interestingly, Q-learningneeds needsthethewidest widest range of values range for for of values all traffic configurations, all traffic configura- requiring a smaller ρ as soon as variation increases. It is worth noting tions, requiring a smaller ρ as soon as variation increases. It is worth noting that that the large therange large of values ([1.9, 6.4]) makes it more difficult to adjust ρ for different traffic configurations. range of values ([1.9, 6.4]) makes it more difficult to adjust ρ for different traffic configu- Conversely, rations. D3QN and Conversely, TD3and D3QN show a smaller TD3 range and show aρsmaller opposite ρ range and behavior opposite as ρ increases behavior as ρ with the traffic variance. increases with the traffic variance. The detailed The detailed evolution evolution ofof the the optimal optimal ρ ρ with with respect respect to to the the traffic traffic variation variation isis plotted plotted in Figure 11a, and Figure 11b shows the total overprovisioning introduced by every in Figure 11a, and Figure 11b shows the total overprovisioning introduced by every RLRL method operating method operating with with the the optimal optimal ρ. We observe ρ. We observe that that overprovisioning overprovisioning increases increases with with traffic variation, traffic variation, with with aa slightly slightly different different trend trend depending depending on the traffic on the traffic pattern pattern (sinusoidal (sinusoidal and real). Moreover, every RL method introduces different amounts of overprovisioning as and real). Moreover, every RL method introduces different amounts of overprovisioning shown in Figure 12, where the relative overprovisioning per RL method with respect to the as shown in Figure 12, where the relative overprovisioning per RL method with respect minimum one for every traffic configuration is represented. Q-learning is the method that to the minimum one for every traffic configuration is represented. Q-learning is the requires larger overprovisioning in general terms, whereas D3QN and TD3 show better method that requires larger overprovisioning in general terms, whereas D3QN and TD3 performance. Interestingly, the differences are proportionally larger when traffic variation show better performance. Interestingly, the differences are proportionally larger when is small. traffic variation is small. Let us analyze the effect in terms of extra-overprovisioning when ρ is fixed to a con- stant value, high enough to assure reliable QoS performance in online RL operation under traffics with a wide range of characteristics. The squared purple markers in Figure 10a indicate the QoS that could be achieved under different traffic characteristics for the largest ρ. Table 3 summarizes the relative and absolute increment of overprovisioning (computed in total Tb per day of operation) produced with the most conservative ρ configuration for all traffic configurations and RL methods. When the fixed ρ happens to be the optimal one for the traffic and RL method, no additional overprovisioning is set up (values in boldface); otherwise, additional overprovisioning is introduced. Q-learning is the method that adds the largest extra overprovisioning (exceeding 200% and 30 Tb/day). In any case, it is worth highlighting that achieving optimal performance in terms of QoS assurance while achieving efficient capacity allocation requires some method to find the optimal ρ for the traffic characteristics. Sensors 2021, 21, 8306 Sensors 2021, 21, 8306 18 of 24 Sensors 2021, 21, 8306 18 of 24 7 20 Q-learning [sin] Q-learning [real] 18 (ρ) D3QN [sin] D3QN [real] Overprovisioning [Tb/day] 6 7 20 TD3 [sin] TD3 [real] Multiplier 16 Q-learning [sin] Q-learning [real] 18 (ρ) 56 14 D3QN [sin] D3QN [real] Overprovisioning [Tb/day] TD3 [sin] TD3 [real] Real Multiplier 16 45 (a) 12 (b) 14 Real 10 Margin (a) 12 (b) 34 8 Sinusoidal 10 Margin 3 6 Sinusoidal Optimal 2 8 2 6 4 Optimal 1 4 2 1 2 0 0 0 0 5 10 15 20 250 0 5 10 15 20 0 5 10 Traffic 15 variation 20 [%] 25 0 5 10 15 variation Traffic 20 [%] 25 Traffic variation [%] Traffic variation [%] Figure Figure 11. Figure11. 11. Optimal Optimal Optimal margin margin margin multiplier multiplier multiplier (a) (a) (a)overprovisioning andand and overprovisioning overprovisioning (b). (b). (b). 40% 40% [sinusoidal] Q-learning [sinusoidal] Q-L Q-learning Q-L 30% D3QN D3QN 30% [%] overprovisioning[%] TD3 20% TD3 20% overprovisioning 10% 10% 0% 0% 40% [real] 40% 30% [real] 30% 20% Extra 10% 20% Extra 0% 10% 1 3 6 12 25 0% Traffic variation [%] 1 3 6 12 25 Figure 12. Relative extra overprovisioning. Traffic variation [%] Figure Relative Figure12.12. extraextra Relative overprovisioning. overprovisioning. Let us analyze the effect in terms of extra-overprovisioning when ρ is fixed to a con- stant value, high enough to assure reliable QoS performance in online RL operation under Table 3. Additional overprovisioning when fixing ρ conservatively. Letwith traffics us analyze the effect a wide range in terms of The of characteristics. extra-overprovisioning squared purple markers when ρ is fixed in Figure 10a t indicate stant the QoS value, Traffic Traffic that could high enough be achieved to assure under different traffic reliable QoS performance Q-Learning D3QN characteristics in online for the TD3 RL operatiolarg- Variance est ρ. Table Pattern 3 summarizes the relative and absolute increment traffics with a wide range of characteristics. The squared purple markers % Tb/Day % Tb/Dayof overprovisioning % Tb/Day (com-in Fig puted in total Tb per day of operation) produced with the most conservative ρ configura- indicate the QoS 1% that could be achieved 0.00% 0 under 0.88%different 0.02 traffic characteristics 2.49% 0.05 for t tion for all traffic 3% configurations and RL 0.67 19.18% methods. When the 6.04% 0.16fixed20.67% ρ happens 0.51to be the est ρ. Table optimal 3 summarizes the relative one for the6%traffic and RL method, and absoluteoverprovisioning no additional increment of overprovisionin Sinusoidal 38.71% 1.85 5.21% 0.21 15.38% is 0.63set up (val- puted ues in in total Tbotherwise, boldface); per day additional 12% of 44.15% operation) 3.56produced0.00% with overprovisioning the most is 0.00 4.01%conservative introduced. 0.32 is the Q-learning ρ co tion methodfor that all traffic 25% adds the 60.40% configurations 8.29 and RL methods. largest extra overprovisioning 0.00% 0.00 When200% (exceeding 0.00% the and fixed 0.00 30 ρ happens Tb/day). In t optimal any case,oneit is for worththehighlighting 1% traffic andthat RLachieving 0.00% method, optimal 0 no additional 0.05% 0.00 performanceoverprovisioning in terms of QoSis 4.95% 0.11 as-set surance while 3% achieving 106.68% efficient capacity3.94 9.52%requires allocation 0.28 some0.39% method 0.01 to find the uesReal in boldface); 6% otherwise,192.50% additional 9.52overprovisioning 12.72% 0.60 is introduced. 2.90% 0.14 Q-learnin optimal ρ for method that adds the traffic 12% characteristics. the largest extra overprovisioning 200.69% 18.37 0.00% (exceeding 0.00 0.00% 200% and 30 Tb/ 0.00 any 25% highlighting case, it is worth 219.35%that34.46 achieving0.00%optimal 0.00 performance 0.00% 0.00 in terms of Table 3. Additional overprovisioning when fixing ρ conservatively. surance while achieving efficient capacity allocation requires some method to To conclude Traffic this section,Q-Learning Traffic we can highlight (i) that D3QN the CRUX problem proposedTD3 in optimal Section ρ for 2 can the traffic be tackled using characteristics. RL; under different traffic characteristics and RL methods, the Pattern Variance % Tb/Day % Tb/Day % Tb/Day target QoS is assured 1% with reduced 0.00% overprovisioning; 0 (ii) online RL 0.88% operation2.49% 0.02 leads to traffic 0.05 Table 3. Additional loss at the beginning ofoverprovisioning flow capacity when operation; fixing this ρ fact conservatively. prevents us from using RL until 3% 19.18% 0.67 6.04% 0.16 20.67% 0.51 online learning has Sinusoidal 6%come up with a robust 38.71% 1.85 and reliable 5.21% model to guarantee 0.21 autonomous 15.38% 0.63 TD3 flowTraffic Traffic comparison operation; and Q-Learning D3QN 12%(iii) the44.15% of the RL 0.00% 3.56 methods shows 0.00interesting differences 4.01% 0.32 Pattern among Variance them. While Q-learning % fast, Tb/Day learns it also produces % overprovisioning. larger Tb/Day D3QN% T 25% 60.40% 8.29 0.00% 0.00 0.00% 0.00 and TD3 need more1% time to ensure 0.00% zero loss and adjust QoS 0 0.05% 0.88% 0.00 at the benefit 0.02 of reducing 2.49% 1% 0.00% 0 4.95% 0.11 overprovisioning. 3% 19.18% 0.67 6.04% 0.16 20.67% 3% 106.68% 3.94 9.52% 0.28 0.39% 0.01 Sinusoidal Real 6%6% 38.71% 192.50% 9.52 1.85 12.72% 5.21% 0.60 0.21 2.90% 15.38% 0.14 12% 12% 44.15% 200.69% 18.37 3.56 0.00% 0.00% 0.00 0.00 0.00% 4.01% 0.00 25% 25% 219.35% 60.40% 34.46 8.29 0.00% 0.00% 0.00 0.00% 0.00 0.00 0.00% 1% 0.00% 0 0.05% 0.00 4.95% mous flow operation; and (iii) the comparison of the RL methods shows interesting dif ferences among them. While Q-learning learns fast, it also produces larger overprovision ing. D3QN and TD3 need more time to ensure zero loss and adjust QoS at the benefit o reducing overprovisioning. Sensors 2021, 21, 8306 19 of 24 5.2. Offline Leaning + Online RL-Based Operation Let us now focus on evaluating the operational approach detailed in Section 4 (Figur 5.2. 2b),Offline whichLeaning consists + Online of threeRL-Based phases.Operation Before emulating flow operation, we generated syn Letdata thetic us nowfor allfocus the on evaluating traffic variancesthe following operationaltheapproach detailed sinusoidal in pattern traffic Section 4and used (Figure 2b), which consists of three phases. Before emulating them to pre-train generic models independently for each traffic configurationflow operation, we gen- and RL erated synthetic data for all the traffic variances following the sinusoidal traffic pattern and method. We ran every offline RL training for 14,400 episodes to guarantee QoS assuranc used them to pre-train generic models independently for each traffic configuration and RL with minimum overprovisioning. method. We ran every offline RL training for 14,400 episodes to guarantee QoS assurance Starting with with minimum the analysis of Phase I, Figure 13a shows an illustrative example of the overprovisioning. evolution Starting ofwith the capacity the analysisduring the I,operation of Phase Figure 13aofshows the threshold-based algorithm an illustrative example of (Algo the evolution of the capacity during the operation of the threshold-based algorithm capacity rithm 4) for the real traffic pattern and variation 12%. Actual traffic x(t), allocated z(t), and the (Algorithm 4) evolution for the realof the self-tuned traffic pattern andkvariation parameter areActual 12%. also shown. Note traffic x(t), that k quickly allocated evolves z(t), capacity fromand its the evolution initial value (k of =the to reach a kcapacity 4) self-tuned parameter are also closer shown. to actual Note However, traffic. that a ksoon quickly evolves from its initial value (k = 4) to reach a capacity closer to as the load exceeds the optimal one l*, k is increased until reaching a stable valu actual traffic. However, as soon as the load exceeds the optimal one l*, k is increased until reaching (1.47), which happens after 60 minutes of operation. The inset table in Figure 13a detail a stable value (1.47), which happens after 60 minutes of operation. The inset table in the values of k after one hour of operation for all traffic configurations. It is worth noting Figure 13a details the values of k after one hour of operation for all traffic configurations. It that is the noting worth self-tuned threshold-based that the algorithm operates self-tuned threshold-based algorithmwith zerowith operates traffic zeroloss for all th traffic cases. loss for all the cases. 100 4.0 30% 90 (a) var k (b) 24.97% 3.5 1% 1 Traffic and Capacity [Gb/s] 25% 80 x(t) 3% 1 3.0 70 z(t) 6% 1.04 20% Traffic Variation 60 k 2.5 12% 1.47 50 25% 1.81 2.0 k 15% 12.56% 40 12% 1.5 10% 30 6.31% 1.0 20 6% 5% 3.27% 10 0.5 3% 1.72% 1% 0 0.0 0% 0 20 40 60 80 10 30 50 70 90 110 Time since path set-up [mm] Time since path set-up [mm] Figure 13.Phase Figure13. PhaseI: Self-tuned threshold I: Self-tuned (a) and threshold (a) traffic variance and traffic analysis variance (b). analysis (b). Parallel to the threshold-based operation, traffic variance analysis (Algorithm 2) is Parallel to the threshold-based operation, traffic variance analysis (Algorithm 2) i conducted in order to compute the true variance of the traffic. Figure 13b shows the conducted in order computed traffic to compute variation the trueofvariance as a function time for ofallthe traffic. traffic Figure 13b shows configurations. Roundthe com puted traffic markers variation highlight when theas aderivative function ofoftraffic time for all traffic variance reachedconfigurations. Round a small eps = 0.01 and marker highlight when the derivative of traffic variance reached a small eps labels show the final computed variance value. The low error between computed and true = 0.01 and labels show the final is variances computed noticeable.variance value. The Such estimation low error is achieved between within two hourscomputed in all theand true variance cases. After two hours is noticeable. of operation,isPhase Such estimation II (Algorithm achieved within 5)twocanhours start and anthe in all RL method cases. with a generic pre-trained After two hoursmodel for the true of operation, traffic Phase variance is set II (Algorithm to operate. 5) can start and Theantuning of RL method with parameter ρ (∆ρ = 0.1) and the resulting QoS are shown in Figure 14a,b, a generic pre-trained model for the true traffic variance is set to operate. The tuning orespectively, for traffic variance equal to 12%. To detect whether the measured QoS is considerably below parameter ρ (∆ρ = 0.1) and the resulting QoS are shown in Figures 14a and b, respectively or above the desired qa value, a significance level cfl = 0.05 was used to be compared for traffic against variancep-value the obtained equal from to 12%. To detecttests. the binomial whether the measured We observe QoS is that ρ decreases upconsiderably to a below or above magnitude the desired that produces QoSqa value, below a significance 99%; level just after that, cfl = 0.05and ρ increases wasremains used tostable be compared against from thatthe pointobtained p-value on. As shown in from Figurethe 14a,binomial the time totests. We observe converge to the bestthat ρ isρ3960, decreases 1440, up to a and 3600 min (2.75, 1, and 2.5 days) for Q-learning, D3QN, and TD3, respectively. The above analysis, however, needs to be complemented with the overprovisioning to extract meaningful conclusions. Figure 15 presents the overprovisioning obtained by every RL method before and after tuning ρ in Phase II. For reference purposes, the overprovisioning introduced by the threshold-based algorithm during Phase I is also included as a dotted line. The large benefits in terms of overprovisioning reduction for the RL-based operation with regard to the threshold-based algorithm are remarkable—up to 45% of capacity allocation reduction and 11 Tb/day of total capacity savings for one 100% 100% 0 1000 2000 3000 3960 4000 5000 QoS 99% Sensors 2021, 21, 8306 20 of 24 (b) s 2021, 21, 8306 98% 0 1000 2000 3000 4000 5000 single flow. After ρ tuning, D3QN Phaseshows II timethe worst performance, as Q-learning and TD3 [mm] achieved significantly lower overprovisioning (24%, ~3 Tb/day). Figure 15 also shows magnitude that produces the obtained overprovisioning Figure 14. Phase II:IIIQoS when the QoS below specific 99%; just model (trained offlineafter with thethat, ρ increa collected traffic) is loaded in Phase after (a) and 10 days ofρoperation. (b) evolution. We observe that Q-learning and from thatoverprovisioning TD3 reduce point on. As shown slightly, in Figure whereas 14a, the a larger reduction time to is achieved withconverge D3QN; to we conclude and 3600 that the(2.75, former 1,RL and methods 2.5aredays) less dependent on an accurate model of the Themin above analysis, however, specific traffic to achieve optimal capacity allocation. for needsQ-learning, D3QN, to be complemented and TD w to extract meaningful conclusions. Figure 15 presents the overp 2 every RL method before and after tuning ρ in Phase II. For refer 1.8 provisioning 1.6 introduced by the threshold-based Q-learning algorithm durin 1440 D3QN as a1.4dotted line. The large benefitsTD3 in terms of overprovisioning re ρ operation 1.2 (a) with regard to3600 the threshold-based algorithm are rem 1 capacity allocation reduction and 11 Tb/day of total capacity sav 100% 100% 0 1000 2000 3000 39604000 5000 After ρ tuning, D3QN shows the worst performance, as Q-learnin nificantly lower overprovisioning (24%, ~3 Tb/day). Figure 15 a QoS 99% overprovisioning when the specific model (trained offline with (b) loaded in Phase III after 10 days of operation. We observe that Q-l 98% overprovisioning 0 1000 slightly, 2000 whereas 3000 4000a larger 5000 reduction is achieve clude that the former PhaseRL II time [mm] methods are less dependent on an accur traffic Figure Figure 14.toPhase achieve 14. Phase optimal II: QoS (a) and II: QoS capacity ρ (b) evolution. (a) and allocation. ρ (b) evolution. 30 The above analysis, however, needs to be complemented wit Q-Learning to extract 25 meaningful conclusions. Figure D3QN 15 presents the overpr Overprovisioning [Tb/day] TD3 every 20 RL method before and Threshold-based after tuning ρ in Phase II. For refere provisioning introduced ~45% by the threshold-based algorithm during ~11 Tb/day as a dotted 15 line. The large benefits in terms of overprovisioning red ~24% ~3 Tb/day operation 10 with regard to the threshold-based algorithm are rem capacity allocation reduction and 11 Tb/day of total capacity savi After5ρ tuning, D3QN shows the worst performance, as Q-learning nificantly 0 lower overprovisioning (24%, ~3 Tb/day). Figure 15 al overprovisioning whenPhase Phase II (Before tuning) theII (After specific tuning) model (trained Phase III offline with Phase loaded in Phase III after 10 days of operation. We observe that Q-le Figure 15. Overprovisioning reduction. overprovisioning Figure 15. Overprovisioning slightly, whereas reduction.a larger reduction is achieved clude Finally, thatletthe us analyze former the performance RL methods of Algorithm are less6 to detect traffic changes dependent on while an accura flow is operated in Phase III; recall that such detection immediately triggers Phase II. To this Finally, traffic to achieve end, let us we generated analyze optimal four the capacity different performance allocation. scenarios, combining two of Algorithm different types of6 to det flow is changes operated in traffic variancein Phase while keepingIII;traffic recall that profile such We unchanged. detection immedia evaluate gradual and30sudden/increase or decrease traffic variance changes. Figure 16 illustrates two out of four this end, scenarios: weincrease gradual generated (variance four different gradually scenarios, increases from combining 1% to 25% along 5 days) andtwo d Q-Learning in 25traffic sudden variance increasing (from 1% to while keeping 25% in just traffic one minute); profile an inverse trendunchanged. is configured for We e ning [Tb/day] gradual and sudden decrease scenarios. D3QN den/increase or decrease traffic variance Threshold-based TD3changes. Figure 16 illustra 20 narios: gradual increase (variance gradually increases from 1% to ~45% sudden 15 increasing (from 1% to 25% ~11 Tb/day ~24% in just one minute); an invers Sensors 2021, 21, 8306 Sensors 2021, 21, 8306 21 of 24 50 (a) 25 Traffic [Gb/s] 0 50 (b) 25 0 0 ,0 2,000 4,000 6,000 8,000 10,000 Time [mm] Figure 16. Figure 16. Phase Phase III: Traffic III: Traffic variancevariance change change scenarios. scenarios. Gradual Gradual increase increase (a) and sudden (a)(b). increase and sudd Table 4 details the time to detect the traffic change when the variance range was Tableas4[var_l, configured details the=time var_h] to+10%]; [−10%, detectthethe traffic current change traffic when variance the variance ra and minimum reward rw_l figured was set var_h] as [var_l, to 5% of=the minimum [−10%, +10%];observed reward (see the current Algorithm traffic 6). We variance and min observe that the proposed mechanism ensures prompt reaction under any of the studied rw_l was set to 5% of the minimum observed reward (see Algorithm 6). We changes—immediate detection is achieved when a sudden change happens, and no more the thanproposed mechanism one hour is required ensures for gradual changeprompt detection. reaction under any of the studi immediate detection is achieved when a sudden change happens, and no m Table 4. Phase III: Analysis under traffic changes. hour is required for gradual change detection. Reward Degradation Traffic Change Detection QoS at Detection Time (%) (Min from Detection) TableScenario 4. Phase Time (min) III: Analysis under traffic changes. Q-L D3QN TD3 Q-L D3QN TD3 45 99.30 100 Gradual increase Gradual QoS at100 Detection Detection419 Time 595 585 Reward De Traffic Change 650 99.86 99.44 99.72 3354 2440 2233 decrease Time (%) (Min from Step Scenario increase 1 99.17 99.86 99.86 332 413 433 Step decrease 1 (min) Q-L 99.44 99.03 99.44 D3QN494 TD3 683 Q-L 212 D3Q Gradual increase To evaluate the promptness 45of detection,99.30 100 the observed Table 4 considers 100 QoS419 at the 59 detection Gradual time, as well as the elapsed decrease 650 time between 99.86 detection 99.44 time 99.72 and the time3354 when 244 reward begins to degrade (reveals whether the RL module is working properly). Note that Step increase the detection happens when the QoS 1 is still above 99.17the target 99.86 99.86 value in all 332 is the cases. This 41 proofStep decrease of anticipation of the change 1 99.03is key 99.44 detection, which to guarantee99.44 494 robust and reliable 68 RL-based operation. To evaluate 6. Concluding the Remarks promptness of detection, Table 4 considers the observ detection The Flowtime, as well Capacity as theOperation Autonomous elapsed(CRUX)time problem between has detection time been introduced to and th deal with online capacity allocation of traffic flows subject to dynamic traffic changes; it reward begins to degrade (reveals whether the RL module is working prope guarantees precise QoS requirement assurance and minimizes capacity overprovisioning. the detection RL-based happens operation when was proposed the QoS to learn is policy the best still above the target for the traffic valueand characteristics in all the proof of anticipation QoS requirements of a givenof flow. the change RL allows detection, adaptive and which is key proactive to guarantee capacity allocation robu once the optimal policy is learnt. However, pure RL operation lacks robustness during RL-based operation. online learning (e.g., at the beginning of flow operation and in the event of traffic changes) and might result in undesirable traffic loss. However, this can be avoided using simpler 6. Concluding reactive Remarks threshold-based capacity allocation. In view of the above, an offline + online learning lifecycle was proposed, aiming The Flow at providing Capacity guaranteed Autonomous performance during the Operation (CRUX) entire lifetime problem of the traffic has been flow. The deal with proposed online capacity management allocation lifecycle consists of phases. of three traffic Firstly, flowsasubject self-tunedtothreshold- dynamic traf based approach was proposed to operate just after the flow guarantees precise QoS requirement assurance and minimizes capacity is set up and until enough over RL-based operation was proposed to learn the best policy for the traffic chara QoS requirements of a given flow. RL allows adaptive and proactive capac once the optimal policy is learnt. However, pure RL operation lacks robu Sensors 2021, 21, 8306 22 of 24 evidence of the traffic characteristics are available (Phase I). Secondly, an RL operation based on models with a pre-trained generic traffic profile but meeting specific traffic variance that was measured during Phase I was executed (Phase II). Lastly, an RL operation with models trained for the real measured traffic, while allowing an online RL to adapt to smooth traffic changes (Phase III). In addition, during Phase III online traffic analysis and RL performance tracking was conducted to detect sharper traffic changes that might require moving back to Phase II to keep high reliability. The proposed lifecycle was implemented under three different RL models, namely, Q- learning, D3QN, and TD3. While Q-learning allows for simple and easy-to-learn definition of policies under discrete spaces of states and actions, D3QN and TD3 enable the application of more complex policies based on deep learning models with continuous state space (D3QN) and continuous action space (TD3). Numerical evaluation of the proposed offline + online lifecycle under different RL techniques was carried out, reproducing realistic traffic flows in a simulation environment. For benchmarking purposes, comparative results against basic threshold-based operation and online RL operation were also presented. The main conclusions extracted from the numerical evaluation are summarized in Table 5, where colors are used to highlight the results. As expected, online RL produces moderate to high loss (reaching peaks of 1–10 Gb/s) at the beginning of the network operation. Among the different methods, Q-learning reached the required QoS operation earlier (up to 6 times faster than TD3) at the expense of moderate to large overprovisioning (up to 40% larger than TD3). On the other hand, D3QN and TD3 needed more time to converge to the required QoS operation but resulted in considerably better capacity allocation efficiency. Table 5. Summary of results for policy-based and RL operation with and without offline learning. Threshold- Approach Concept Q-Learning D3QN TD3 Based Traffic loss Zero traffic loss Policy-based Since path QoS assurance set-up Over-provisioning Very large Traffic loss Moderated loss Moderated loss High loss QoS assurance After 2 days After 2 days After 5 days Online RL ρ range for QoS assurance Wide Narrow Narrow Operation Over- conservative ρ Large Small Small provisioning optimal ρ Moderate Small Small Traffic loss Zero traffic loss Zero traffic loss Zero traffic loss Zero traffic loss Since path Since path Since path Since path QoS assurance set-up set-up set-up set-up Offline + ρ fine tuning effectiveness Large Moderated Large Online RL Phase I None Operation Over- Phase I → Phase II Moderated Large Large provisioning Phase II Large Small Large Gain Phase II → Phase III Small Large Small Reliability (Phase III → Phase II) High High High The analysis of the numerical results of the proposed lifecycle leads to several conclu- sions. Firstly, zero traffic loss and QoS assurance is guaranteed from path set-up regardless of the chosen RL method. Secondly, Phase II allows a very efficient and robust operation based on pre-trained generic models that were tuned with specific traffic characteristics. Phase II clearly outperformed threshold-based operation in terms of capacity utilization since it remarkably reduced overprovisioning (up to 45%). Thirdly, all the methods reached outstanding capacity efficiency (more than 50% of capacity reduction with respect to threshold-based operation) without losing QoS performance in Phase III; Q-learning and TD3 behaved slightly better than D3QN. Finally, the continuous analysis and tracking conducted during Phase III to detect traffic changes allowed a prompt detection of sharp Sensors 2021, 21, 8306 23 of 24 changes (between 1 and 650 minutes), triggering Phase II from several hours to days before online RL operation suffered any significant degradation. Author Contributions: Conceptualization, L.V.; methodology, M.R.; software and validation, S.B.; writing—original draft preparation, S.B.; writing—review and editing, M.R. and L.V. All authors have read and agreed to the published version of the manuscript. Funding: This research received funding from the European Community through the B5G-OPEN project (101016663), from the AEI IBON project (PID2020-114135RB-I00), and from the ICREA Institution. Institutional Review Board Statement: Not applicable. Informed Consent Statement: Not applicable. Data Availability Statement: The data presented in this study are available on request from the corresponding author. Conflicts of Interest: The authors declare no conflict of interest. References 1. Neves, P.; Calé, R.; Costa, M.; Gaspar, G.; Alcaraz-Calero, J.; Wang, Q.; Nightingale, J.; Bernini, G.; Carrozzo, G.; Valdivieso, Á.; et al. Future mode of operations for 5G–The SELFNET approach enabled by SDN/NFV. Comput. Stand. Interfaces 2017, 54, 229–246. [CrossRef] 2. Velasco, L.; Piat, A.C.; Gonzlez, O.; Lord, A.; Napoli, A.; Layec, P.; Rafique, D.; D’Errico, A.; King, D.; Ruiz, M.; et al. Monitoring and Data Analytics for Optical Networking: Benefits, Architectures, and Use Cases. IEEE Netw. 2019, 33, 100–108. [CrossRef] 3. Bonati, L.; D’Oro, S.; Polese, M.; Basagni, S.; Melodia, T. Intelligence and Learning in O-RAN for Data-Driven NextG Cellular Networks. IEEE Commun. Mag. 2021, 59, 21–27. [CrossRef] 4. Clemm, A.; Ciavaglia, L.; Granville, L.; Tantsura, J. Intent-Based Networking—Concepts and Definitions. IRTF 2021. Draft, Work-in-Progress. 5. Velasco, L.; Signorelli, M.; De Dios, O.G.; Papagianni, C.; Bifulco, R.; Olmos, J.J.V.; Pryor, S.; Carrozzo, G.; Schulz-Zander, J.; Bennis, M.; et al. End-to-End Intent-Based Networking. IEEE Commun. Mag. 2021, 59, 106–112. [CrossRef] 6. Rafique, D.; Velasco, L. Machine Learning for Network Automation: Overview, Architecture, and Applications [Invited Tutorial]. J. Opt. Commun. Netw. 2018, 10, D126–D143. [CrossRef] 7. Sutton, R.S.; Barto, A.G. Reinforcement Learning: An Introduction, 2nd ed.; MIT Press: Cambridge, MA, USA, 2018; p. 1054. 8. Wang, Z.; Schaul, T.; Hessel, M.; Hasselt, H.; Lanctot, M.; Freitas, N. Dueling Network Architectures for Deep Reinforcement Learning. In Proceedings of the International Conference on Machine Learning, New York, NY, USA, 20–22 June 2016. 9. Fujimoto, S.; Hoof, H.; Meger, D. Addressing Function Approximation Error in Actor-Critic Methods. In Proceedings of the International Conference on Machine Learning, Stockholm, Sweden, 10–15 July 2018. 10. Velasco, L.; Gifre, L.; Izquierdo-Zaragoza, J.L.; Paolucci, F.; Vela, A.P.; Sgambelluri, A.; Ruiz, M.; Cugini, F. An Architecture to Support Autonomic Slice Networking. J. Lightw. Technol. 2018, 36, 135–141. [CrossRef] 11. Velasco, L.; Ruiz, M. Supporting Time-Sensitive and Best-Effort Traffic on a Common Metro Infrastructure. IEEE Commun. Lett. 2020, 24, 1664–1668. [CrossRef] 12. Van Hasselt, H.; Guez, A.; Silver, D. Deep Reinforcement Learning with Double Q-learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 12–17 February 2016. 13. Chen, X.; Li, B.; Proietti, R.; Lu, H.; Zhu, Z.; Ben Yoo, S.J. DeepRMSA: A Deep Reinforcement Learning Framework for Routing, Modulation and Spectrum Assignment in Elastic Optical Networks. J. Light. Technol. 2019, 37, 4155–4163. [CrossRef] 14. Chen, X.; Proietti, R.; Ben Yoo, S.J. Building Autonomic Elastic Optical Networks with Deep Reinforcement Learning. IEEE Commun. Mag. 2019, 57, 20–26. [CrossRef] 15. Troia, S.; Sapienza, F.; Vare, L.; Maier, G. On Deep Reinforcement Learning for Traffic Engineering in SD-WAN. IEEE J. Sel. Areas Commun. 2021, 39, 2198–2212. [CrossRef] 16. Velasco, L.; Barzegar, S.; Sequeira, D.; Ferrari, A.; Costa, N.; Curri, V.; Pedro, J.; Napoli, A.; Ruiz, M. Autonomous and Energy Efficient Lightpath Operation Based on Digital Subcarrier Multiplexing. IEEE J. Sel. Areas Commun. 2021, 39, 2864–2877. [CrossRef] 17. Barzegar, S.; Ruiz, M.; Velasco, L. Reinforcement Learning-based Autonomous Multilayer Network Operation. In Proceedings of the 2020 European Conference on Optical Communications (ECOC), Brussels, Belgium, 6–10 December 2020; pp. 1–4. 18. Panayiotou, T.; Manousakis, K.; Chatzis, S.P.; Ellinas, G. A Data-Driven Bandwidth Allocation Framework With QoS Considera- tions for EONs. J. Light. Technol. 2019, 37, 1853–1864. [CrossRef] 19. Velasco, L.; Shariati, B.; Boitier, F.; Layec, P.; Ruiz, M. Learning Life Cycle to Speed Up Autonomic Optical Transmission and Networking Adoption. J. Opt. Commun. Netw. 2019, 11, 226–237. [CrossRef] 20. Ruiz, M.; Ruiz, M.; Tabatabaeimehr, F.; Gifre, L.; López-Buedo, S.; de Vergara, J.L.; González, Ó.; Velasco, L. Modeling and Assessing Connectivity Services Performance in a Sandbox Domain. J. Lightw. Technol. 2020, 38, 3180–3189. [CrossRef] Sensors 2021, 21, 8306 24 of 24 21. Velasco, L.; Barzegar, S.; Tabatabaeimehr, F.; Ruiz, M. Intent-Based Networking for Optical Networks [Invited Tutorial]. IEEE/OSA J. Opt. Commun. Netw. 2022, 14, A11–A22. [CrossRef] 22. Powell, W. Approximate Dynamic Programming, 2nd ed.; Wiley: Hoboken, NJ, USA, 2011. 23. Mahmoud, H. Parametric versus Semi and Nonparametric Regression Models. Int. J. Stat. Probab. 2021, 10, 90–109. [CrossRef] 24. Brockwell, P.; Davis, R. Introduction to Time Series and Forecasting; Springer: Berlin/Heidelberg, Germany, 2016. 25. Casella, G.; Berger, R. Statistical Inference; Duxbury Press: Pacific Grove, CA, USA, 2002. 26. Ruiz, M.; Coltraro, F.; Velasco, L. CURSA-SQ: A Methodology for Service-Centric Traffic Flow Analysis. J. Opt. Commun. Netw. 2018, 10, 773–784. [CrossRef] 27. Bernal, A.; Richart, M.; Ruiz, M.; Castro, A.; Velasco, L. Near real-time estimation of end-to-end performance in converged fixed-mobile networks. Comput. Commun. 2020, 150, 393–404. [CrossRef]