Radiotherapy and Oncology 153 (2020) 26–33 Contents lists available at ScienceDirect Radiotherapy and Oncology journal homepage: www.thegreenjournal.com Review Article What is plan quality in radiotherapy? The importance of evaluating dose metrics, complexity, and robustness of treatment plans Victor Hernandez a,1,⇑, Christian Rønn Hansen b,c,d,1, Lamberto Widesott e,1, Anna Bäck f,g, Richard Canters h, Marco Fusella i, Julia Götstedt j, Diego Jurado-Bruggeman k, Nobutaka Mukumoto l, Laura Patricia Kaplan m, Irena Koniarová n, Tomasz Piotrowski o,p, Lorenzo Placidi q, Ana Vaniqui r, Nuria Jornet s a Department of Medical Physics, Hospital Sant Joan de Reus, IISPV, Spain; b Laboratory of Radiation Physics, Odense University Hospital; c Institute of Clinical Research, University of Southern Denmark; d Danish Centre for Particle Therapy, Aarhus University Hospital, Denmark; e Centro di protonterapia, APSS, Trento, Italy; f Department of Therapeutic Radiation Physics, Medical Physics and Biomedical Engineering, Sahlgrenska University Hospital, Gothenburg; g Department of Radiation Physics, Institute of Clinical Sciences, Sahlgrenska Academy at the University of Gothenburg, Sweden; h Department of Radiation Oncology (Maastro), GROW School for Oncology, Maastricht University Medical Centre+, The Netherlands; i Medical Physics Department, Veneto Institute of Oncology IOV – IRCCS, Padua, Italy; j Department of Radiation Physics, University of Gothenburg, Göteborg, Sweden; k Medical Physics and Radiation Protection Department, Institut Català d’Oncologia, Girona, Spain; l Department of Radiation Oncology and Image-applied Therapy, Graduate, School of Medicine, Kyoto University, Japan; m Department of Oncology, Aarhus University Hospital, Denmark; n National Radiation Protection Institute, Prague, Czech Republic; o Department of Electroradiology, Poznań University of Medical Sciences; p Department of Medical Physics, Greater Poland Cancer Centre, Poznań, Poland; q Fondazione Policlinico Universitario ‘‘A. Gemelli” IRCCS, UOC Radioterapia Oncologica, Dipartimento di Diagnostica per Immagini, Radioterapia Oncologica ed Ematologia, Roma, Italy; r Department of Radiation Oncology (Maastro), GROW School for Oncology, Maastricht University Medical Centre+, The Netherlands; s Servei de Radiofísica i Radioprotecció, Hospital de la Santa Creu i Sant Pau, Barcelona, Spain a r t i c l e i n f o a b s t r a c t Article history: Plan evaluation is a key step in the radiotherapy treatment workflow. Central to this step is the assess- Received 2 July 2020 ment of treatment plan quality. Hence, it is important to agree on what we mean by plan quality and to Received in revised form 17 September be fully aware of which parameters it depends on. We understand plan quality in radiotherapy as the 2020 clinical suitability of the delivered dose distribution that can be realistically expected from a treatment Accepted 17 September 2020 Available online 25 September 2020 plan. Plan quality is commonly assessed by evaluating the dose distribution calculated by the treatment planning system (TPS). Evaluating the 3D dose distribution is not easy, however; it is hard to fully eval- uate its spatial characteristics and we still lack the knowledge for personalising the prediction of the clin- Keywords: Plan quality ical outcome based on individual patient characteristics. This advocates for standardisation and Dose metrics systematic collection of clinical data and outcomes after radiotherapy. Additionally, the calculated dose Complexity distribution is not exactly the dose delivered to the patient due to uncertainties in the dose calculation Robustness and the treatment delivery, including variations in the patient set-up and anatomy. Consequently, plan Robust optimisation quality also depends on the robustness and complexity of the treatment plan. We believe that future Robust evaluation work and consensus on the best metrics for quality indices are required. Better tools are needed in Complexity metrics TPSs for the evaluation of dose distributions, for the robust evaluation and optimisation of treatment plans, and for controlling and reporting plan complexity. Implementation of such tools and a better understanding of these concepts will facilitate the handling of these characteristics in clinical practice and be helpful to increase the overall quality of treatment plans in radiotherapy. Ó 2020 The Authors. Published by Elsevier B.V. Radiotherapy and Oncology 153 (2020) 26–33 This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/). Evaluation of treatment plans is a key step in the radiotherapy different characteristics of the treatment plan and there is no glo- process that determines the characteristics of the plan selected for bal consensus on how exactly to define, measure, and report plan treatment and, consequently, how patients undergoing radiother- quality. For this reason, the 3rd Physics ESTRO Workshop held in apy are treated. The goal of plan evaluation is to assess a plan’s Budapest in October 2019 included a track on ‘Plan quality assess- quality using a number of qualitative and/or quantitative mea- ment’. The purpose of this work is to summarise the contents dis- sures. The concept of plan quality can, however, encompass many cussed during this Workshop and to provide the participants’ vision on this topic. ⇑ Corresponding author at: Department of Medical Physics, Hospital Sant Joan de This work comprises three sections focused on different aspects Reus, IISPV, Spain. that affect the quality of treatment plans: dose metrics, plan E-mail address: vhernandezmasgrau@gmail.com (V. Hernandez). robustness and plan complexity. Finally, we present the working 1 The first three authors contributed equally to this work. https://doi.org/10.1016/j.radonc.2020.09.038 0167-8140/Ó 2020 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/). V. Hernandez, Christian Rønn Hansen, L. Widesott et al. Radiotherapy and Oncology 153 (2020) 26–33 group’s overall conclusions on the subject of treatment plan qual- ity (TCP) and normal tissue complication probability (NTCP) can be ity assessment. calculated from biological models [13,14], which are usually based on the DVH information. Most of these tolerances and NTCP mod- els were generated using data from one or few centres and should, Dose metrics therefore, be used with caution. If possible, a local validation of the NTCP model should be conducted but, as a minimum, an under- The evaluation of the calculated dose distribution is a funda- standing of the underlying model cohort and the limitations of mental aspect of plan evaluation. It is critical to ensure high plan the model is required [15]. It is also important, when possible, to quality during the treatment planning process and upon treatment minimise OAR doses below model tolerances to facilitate potential approval. However, this evaluation is not easy since there are mul- reirradiations in case of tumour relapse [16]. tiple aspects that commonly play a role in the assessment of plan Derived DVH dose metrics can also be useful to objectively quality. These include protocols, local requirements, historical cus- quantify the quality of a dose distribution regarding target irradia- toms and personal preferences, and they can affect any final tion [17] and have been adopted in radiotherapy guidelines. These decision. metrics include homogeneity, conformity and gradient indices and Prescriptions and treatment planning protocols should incorpo- some of them indirectly ensure low doses to the normal tissues. rate a prioritisation of the dose objectives and constraints on tar- They are used in some protocols that require specific dose distribu- gets and organs at risk (OARs). This prioritisation should be tions (e.g., stereotactic treatments and brachytherapy) and differ- based on expert evaluation of the scientific literature, as done by ent definitions are used depending on the clinical purpose and the DAHANCA group [1,2]. Such a prioritisation would help to stan- the considered protocol, each one with its own strengths and dardise the decision-making process and reduce subjectivity when drawbacks. Homogeneity indices are usually based on ratios of clinical compromises are needed. It is important to be critical two or three points in the target DVH, such as different definitions towards historical customs and traditions, however. These must of maximum, minimum and prescribed dose [7,8]. More complex always be supported by scientific evidence or at least be regularly indices can take into account all the points in the DVH curve and re-evaluated based on new experience. Inter-planner variability quantify the dose dispersion around the average dose [18,19]. Con- may be another factor that affects the final treatment plan and formity indices (CI) involve ratios of volumes treated at a given iso- the resulting dose distribution. This can be reduced by using auto- dose to the target volume [20]. The RTOG CI [21] is criticised for matic planning strategies [3,4]. In any case, personal preferences not taking into account the location and shape of the prescription based on subjective beliefs, such as attempting to make the dose isodose with respect to the target volume. To overcome this limi- distribution ‘look’ nice (for instance prioritising conformity over tation, other indices have been proposed, such as the Paddick CI, dose to OARs and normal tissues), should be avoided. which combines conformity and target coverage [22]. A steep dose To that purpose, the use of class solutions in treatment planning fall-off is important in order to decrease toxicity in tissues sur- is highly recommended and local planning protocols should be rounding the target, especially in treatments involving high doses adapted to each institution’s specific requirements, be evaluated per fraction. Dose gradient indices quantify this property [23]. periodically and incorporate dynamic development [5]. Addition- It can be practical to collapse all this information into an overall ally, it is highly recommended to standardise the collection of plan score or single plan quality index (PQI), as recently pointed patient data such as co-morbidities, systemic treatments, toxici- out by Giglioli et al. [24]. PQIs are obtained by weighing DVH val- ties, local control and survival in a database, regardless of their ues and DVH-derived metrics based on patient group specific pro- inclusion in clinical trials. This will help build clinical knowledge tocols and they facilitate comparisons while minimising based on earlier experiences. The consistency of such data is cru- subjectivity. Large efforts have been made by several authors to cial and any potential sources of bias, such as how the dose distri- define a composite score or figure of merit for quantifying overall bution is normalised [6–8] and what calculation algorithm and plan quality [25–29] and there is a commercial tool available dose quantity is used [9] should be carefully considered and [30,31]. These quality indices are practical as decision-support sys- reported. tems when comparing plans for the same patient [32] and have The evaluation of the calculated dose distribution is often based also been used for evaluating plan quality across different patients on dose volume histograms (DVHs), which collapse the 3D dose and platforms [24] and in treatment planning competitions [33– information in 2D metrics (dose and volume), losing the informa- 35]. However, which parameters to consider and their weights is tion on its spatial distribution. Due to this limitation, a slice-by- not a straightforward decision and involves a certain degree of sub- slice inspection of the dose distribution is recommended to iden- jectivity. Hence, careful evaluation is required in comparisons tify potential aspects for further improvement [7]. The target cov- across different patients and technologies. PQIs are the basis of erage is commonly evaluated by comparing DVH metrics as well as the cost functions used in plan optimisers and in automatic plan- hot and cold regions to protocolised goal values [6–8]. Regarding ning strategies; therefore, these aspects should be carefully consid- OARs, studies on clinical tolerances based on DVH metrics have ered when plans are evaluated and when automatic planning been collected by Emami et. al. [10] and updated in the QUANTEC models are trained [36]. series of publications by Bentzen et al. [11]. These tolerances rely Texture analysis was recently proposed to characterise the spa- on a single DVH endpoint and were based on an arbitrary level of tial dose distribution. This is known as dosiomics [37–42] and is a acceptable toxicity. This can be problematic since dose metrics powerful technique for characterising spatial and statistical distri- are often highly correlated and a single dose metric might not butions of pixel/voxel intensities in an image through the identifi- encapsulate the full causal dose response relationship. Existing cation of patterns and voxel correlations. Dosiomics is a promising mathematical methods such as principal component analysis, method for parameterising regions of interest and for producing which can disentangle the collinearity effects of dose metrics intensity, textural and shape-based dose features that might be [12], can help overcome these problems. Another issue is the fact able to describe the dose distribution better than DVH-based met- that, when QUANTEC data were collected, the use of 3D TPSs was rics, as well as to potentially improve the predictive performance not fully implemented and data analysis capacities of computers of TCP and NTCP models. Moreover, dosiomics can provide novel were limited. quality metrics from the extracted features, facilitating a more Instead of evaluating the target coverage and the risk of toxicity accurate evaluation and further standardisation of treatment plan through a single dosimetric endpoint, the tumour control probabil- quality. Feature extraction methodologies are comparable with 27 Plan quality in radiotherapy: dose metrics, complexity, and robustness those used by radiomics but they involve 3D dose matrices instead OARs in treatment planning [48]. All these concerns about the of diagnostic images (CT, MRI, etc.). Among the radiomic features use of PTV and PRV are even more important in proton therapy. defined in the Image Biomark Standardization Initiative [43], the Robust optimisation follows a different approach, addressing most promising features for dosiomic studies belong to the first uncertainties explicitly, without the need for margins. Thus, order and textural families (local intensity, intensity-based statis- instead of optimising a single scenario (i.e., patient in a fixed nom- tics, grey level co-occurrence matrix, grey level size zone matrix, inal position and considering PTV and PRV margins) dose distribu- etc..). A standardisation and stability analysis of the selected dosio- tions are optimised for n scenarios. Each scenario represents a mic features is mandatory [44], as well as a feature redundancy possible treatment course (i.e. all the treatment fractions) and investigation to select the dosiomic features that are independent should include the specification of all the errors that are needed of DVH-based metrics and which can thus provide additional value. to calculate a final dose distribution. In the context of random Not only anatomy-based ROIs (e.g. delineated target volumes and setup errors in a fractionated treatment, this would also include OAR) should be considered for the dosiomic feature extraction; the setup errors for all individual fractions. The plan can then be instead, dedicated dosiomic regions of interest from which to optimised considering all of the n scenarios at once (often in the extract the features (isodose volumes, intersections of anatomy- worst-case scenario, minimax approach) or a combination of dif- based ROIs, regions of low bath dose, etc.) may offer a better eval- ferent scenarios, each one assigned a certain probability (proba- uation of the 3D dose distribution. There is also a need for dosiomic bilistic approach). features to have a clear clinical interpretation because ‘black box’ Regardless of whether the plan has been optimised in the ’clas- features would be difficult or impossible to prioritise against other sical’ way (i.e., PTV-based) or with robust optimisation, it is always metrics. possible to make an a posteriori assessment of the plan’s robust- Ideally, the quality of a dose distribution should be linked to ness. With robustness evaluation one can evaluate how the dose evidence-proven clinical outcomes and should be personalised to distribution changes compared to the nominal dose by recalculat- account for patient-specific data such as tumour characteristics, ing the plan obtained in different error scenarios. This offers the complementary treatments, and risk factors. However, this is a possibility of quantifying the uncertainties in DVHs and other dose truly complex problem involving a large amount of data and it is metrics due to e.g. variations in patient set-up and anatomy not well known yet how to carry out this personalisation. This pro- [50,51]. vides a compelling challenge not only for TPS manufacturers but It has been shown that robust optimisation can potentially for the whole scientific community. In general, we believe that solve the PTV/PRV limitations and improve CTV coverage and more effort is warranted by: OAR sparing [52–54] both for photon and proton treatments. The problem of PTV expansion into air is also solved since variations  TPS manufacturers, to implement additional tools for evaluating concerning the CTV are explicitly accounted for in the calculation. dose distributions, such as radiobiological models, as well as to Robust optimisation and evaluation are generally posed as n- facilitate the future implementation of new spatially correlated dimensional problems; for instance, n = 1 can take into account tools, such as dosiomics, and by the set-up error, n = 2 the range uncertainty for protons, n = 3  the scientific community, to collect structured data on dose the breathing phases of a 4DCT and n = 4 possible anatomical indices, patient and tumour characteristics and follow-up data changes (such as cavity filling and tumour shrinkage) to reduce on tumour control, toxicities and survival. Future analysis of the need for re-planning. In general, robust optimisation reduces the data with appropriate tools such as Big Data Analytics and doses to normal tissues that would be unnecessarily irradiated artificial intelligence (AI) could help personalise radiotherapy with the PTV-margin concept and leads to delivered dose distribu- treatments and further link the quality of a dose distribution tions with a higher probability of acceptable target coverage and to the patient’s clinical outcome. OAR sparing [52–54]. Only a few years since it was first implemented in a commercial TPS in 2014, robust optimisation has become a standard tool in Robustness proton therapy and is also gaining interest in photon therapy. A treatment plan that is both ‘good’ (i.e., with a clinically acceptable The traditional method to achieve target coverage and OARs nominal dose distribution), and ‘robust’ will yield a dose distribu- sparing has been the definition of adequate margins around the tion that is suitable for all or the majority of the error scenarios clinical target volume (CTV) and OARs to obtain the planning target considered by the optimiser. There are different paradigms to volume (PTV) and organ at risk planning volume (PRV). Several for- translate this notion into mathematical terms. Broadly, these mulas have been proposed for the definition of these margins for approaches can be categorised as (i) the probabilistic (stochastic) the PTV [45,46] and PRVs [47,48]. approach, which optimises the expected delivered dose distribu- However, there are several limitations that affect the PTV defi- tion, and (ii) the minimax approach, which optimises the dose dis- nition: it relies on the so-called static dose cloud approximation tribution for the worst error considered [55]. Probabilistic and does not guarantee optimal management for PTVs extending optimisation requires associating an occurrence probability to each into air. Moreover, whether or not the CTV receives the prescribed considered uncertainty scenario. Gaussian probability densities are dose depends also on the specific dose distribution rather than commonly assumed as a first approximation, which yields treat- only on geometric margins. Dose distributions are neither perfectly ment plans that are also robust against other probability density conformal to the PTV nor equally conformal on all sides of the CTV functions [56]. Worst-case methods can focus on (i) composite and non-conformity results in an inherent dosimetric margin [49]. worst-case scenario (composite worst-case), (ii) worst-case scenar- In those regions where the prescription isodose line extends ios for each objective considered independently (objectivewise beyond the CTV less or no margin needs to be added to account worst-case) and (iii) worst-case scenarios for each voxel consid- for treatment uncertainties. In addition to conformity, the required ered independently (voxelwise worst case) and no particular margin also depends on the steepness of the dose fall-off near the method has yet proven to be superior to the others in all circum- target because a naturally gradual fall-off may require a smaller stances [57]. margin than a sharp fall-off. As Stroom et. al. underlined, the PRV We must bear in mind that the terms ‘robust optimisation’ and concept has even more limitations and it seems necessary to ‘robust analysis’ encompass several different methodologies and develop alternative ways to include geometric uncertainties of metrics, and there is no consensus on which ones should be imple- 28 V. Hernandez, Christian Rønn Hansen, L. Widesott et al. Radiotherapy and Oncology 153 (2020) 26–33 mented or exactly how they should be used [55,58–61]. Robust-  Generate databases of clinical data and robustness parameters ness evaluation tools are in their infancy and many metrics can and share these data among different centres. This will enable be used that provide global (DVHs, variations in dosimetric indices, tumour control and OAR toxicity to be correlated with the etc.) or local (2D dose distribution) information. results of robustness evaluations. A message that clearly emerges from the literature is that there can be significant differences, both in terms of target cov- Plan complexity erage and OAR sparing, between the nominal dose distribution approved by the clinician and the dose distribution in the dif- Modern modulated radiotherapy techniques involve modula- ferent scenarios evaluated in the robustness analysis. This is tion of many machine parameters, placing high demands on treat- true both for photon and proton plans, which is why robustness ment machines and TPSs. This increase in modulation of machine analysis should not be restricted to the proton therapy world parameters is commonly referred to as an increase in treatment alone. plan complexity. More complex plans have larger uncertainties in In order to make robustness analysis clinically useful, it is extre- dose calculation and treatment delivery [63–65] compared to mely important to define a benchmark for the robust analysis tools non-modulated plans. and agree on what constitutes a safe degree of variability. Is it It is challenging to agree on a unique definition of complexity, acceptable, for instance, if in the worst-case scenario the D1% to even among scientists [66]. We understand the complexity of a the spinal cord increases by 20 Gy compared to the nominal dose, radiotherapy treatment plan as an estimation of the degree of dose even though it remains within tolerance? Calibration of robust uncertainty as a result of its calculation and delivery, which evaluation methods against existing PTV-based methods could be depends on all the machine parameters that make up the treat- a solution for this [58]. Another important consideration must be ment plan. made when robustness is applied to OAR dose constraints in addi- Several complexity metrics have been proposed in the literature tion to target coverage. The OAR constraints reported in the QUAN- for MLC-based treatments. Initially, they were based on fluence TEC papers [11] are all based on nominal DVHs and no robust maps, but these maps are not always available. Furthermore, the analysis was considered in the setting of those constraints. There- same fluence maps can be produced through different variations fore, the request that these OAR constraints are fulfilled in many, if of machine parameters depending on the optimisation and not all, the considered scenarios may be too restrictive and com- sequencing algorithms used, which cannot be taken into account promise target coverage. by fluence-based metrics [67]. Therefore, fluence-based metrics Robustness methods address variations in the patient’s anat- have been gradually replaced by metrics that directly depend on omy throughout the course of treatment, but the uncertainty in the treatment plan parameters. Different complexity metrics focus volume delineation is typically not considered. Automatic contour- on different aspects of plan complexity [68], such as aperture mod- ing algorithms that improve inter-observer consistency are cur- ulation [67,69], the size and irregularity of beam apertures [67,70– rently available for clinical use, especially for organs at risk [62]. 73] and modulation of specific machine parameters, e.g. the dis- Volume delineation still constitutes an important potential source tance travelled by the leaves [74] and the variations in dose rate of uncertainty in radiotherapy, however. Interestingly, robustness and gantry speed [75,76]. A detailed explanation of complexity methods could also be applied to delineation uncertainties, but metrics can be found in two recent reviews [77,78]. convincing applications of robust optimisation in this context are A certain degree of complexity in treatment plans is necessary limited [55]. because it is often required to achieve an acceptable dose distribu- Given the potential of these new tools and their current avail- tion. However, many investigators have reported that a high ability in treatment planning systems, it is important to reach con- degree of plan complexity may affect the accuracy of dose calcula- sensus on which methods are the most appropriate for both robust tion and treatment delivery [69,71,72,74,75,79,80]. This is because optimisation and evaluation. Retrospectively analysing plan more complex plans typically involve smaller and more irregular robustness for patients already treated with protons (with follow beam apertures, larger tongue-and-groove effects and larger mod- up data on toxicity and tumour control) could be an interesting ulation of machine parameters. Such complexities affect the uncer- solution. These data could be useful for establishing appropriate tainties in dose calculations due to limitations in the calculation references for robustness parameters, as proposed by Malyapa et. algorithm or in the beam model, e.g. in the MLC configuration al. [61]. We believe the same should be done for typical IMRT [81]. They also influence the sensitivity of the delivered dose to and VMAT plans in different treatment sites in order to enable a small deviations in machine parameters during treatment delivery transition to margin-less robust optimisation and evaluation in (even within their tolerances) and to variations in patient geome- the photon world. try, e.g. respiratory motion [82]. Plan complexity can be inter- In this context we consider it essential to: preted as an assessment of the robustness of treatment plans to all these uncertainties. Additionally, highly complex plans require  Implement more and faster tools in TPSs to facilitate both longer beam-on times, which can increase the risk of intra- robust optimisation and robust evaluation of treatment plans. fraction movements in some treatment sites [83]. A high degree These tools should preferably include not only the worst-case of plan complexity may therefore compromise the overall accuracy scenario, but also the probabilistic approach and be available and quality of the treatment. Consequently, high degrees of plan for both proton and photon planning. complexity should be avoided to maximise accuracy and robust-  Further perform robust optimisation and evaluation of clinical ness of radiation treatments. AAPM pointed out the need to quan- plans, also for photon beam therapy. tify plan modulation [84] and to adapt tolerance limits to the  Establish a common benchmark for plan robustness evaluation degree of plan complexity [65]. that is used for both particle and photon beam therapy and It is known that ‘dosimetric’ plan quality is often not correlated allows a fair comparison of the robustness of proton and photon to plan complexity [85–87] and that similar dose distributions are plans. achievable with more or less complex treatment plans because  Reach consensus in the scientific community on which methods inverse optimisation can introduce unnecessary complexity and metrics are the most appropriate for robust optimisation [70,85,88]. For these reasons, many investigators have recom- and analysis, e.g. to quantify the degree of robustness of a given mended incorporating complexity metrics into the cost function plan. used by optimisation algorithms [69,70,79,86,89]. This is particu- 29 Plan quality in radiotherapy: dose metrics, complexity, and robustness larly relevant in automated planning, where there is an added risk Plan quality of inadvertently increasing plan complexity [90–92]. There is no consensus on which complexity metric should be used. Many of The quality of a radiotherapy treatment plan should indicate its them are correlated and multiple metrics can be used to account clinical suitability, which can be assessed from the delivered dose for the different uncertainties and sources of plan complexity distribution that can realistically be expected. To that purpose, the [68]. A very comprehensive table was recently provided by Antoine existing uncertainties in the calculation and delivery of the plan et al. [78], summarising the main proposed complexity metrics and their impact on the delivered dose must be accounted for. Plan that could be used. Several different metrics may have a similar quality is typically assessed by evaluating to what degree a ‘nom- impact in reducing plan complexity and further studies are needed. inal’ calculated dose distribution fulfils the desired dose objectives In our opinion, the more relevant aspects in plan complexity are specified in the radiation oncologist’s prescription. It is important the degree of aperture modulation and the irregularity of the beam to bear in mind, however, that the dose ‘on the screen’ is not the apertures. Hence, a combination of metrics addressing these two dose actually delivered to the patient. Differences may arise due aspects could be used to control plan complexity during the opti- to limitations in the models and algorithms implemented in treat- misation process. Additionally, it would be desirable that TPSs ment planning systems [99], which can be even more relevant for clearly describe the metrics that they use and report the obtained navigated plans in Pareto fronts using multi-criteria optimisation values to facilitate the management of plan complexity in clinical (MCO)[100,101]. Differences can also be due to uncertainties in practice. the delivery of the treatment plan [102,103], including those asso- Information on the complexity of treatment plans can be help- ciated with the treatment unit as well as with patient set-up, intra- ful to evaluate trade-offs between dosimetric performance and fraction motion, and variations in the patient’s anatomy [104]. plan complexity. Additionally, a plan complexity metric can serve As discussed in the previous sections, the impact of these uncer- as a plan verification tool aimed at reducing the QA workload, tainties on the dose delivered to the patient depends on the robust- e.g. because plans with a low degree of complexity may not ness and complexity of treatment plans. Despite that, plan quality require as much verification as highly complex plans [77,78]. is commonly quantified solely in terms of the calculated dose dis- This can be important for the QA of online adaptive radiotherapy, tribution in a single scenario [25,27–30], with few exceptions where measurement-based verifications are not feasible and where aspects related to plan robustness, complexity, and treat- complexity metrics can facilitate a fast verification of the adapted ment efficiency were considered [59,105–108]. In our opinion, plans [93]. when only the calculated dose distribution in the nominal situa- For the reasons discussed above, plan complexity is a relevant tion is examined this should be clearly acknowledged and aspect when evaluating treatment plan quality. The quality of a reported; in these cases, terms such as ‘dosimetric plan quality’ treatment plan depends not only on the calculated dose distribu- or ‘dosimetric performance’ could be used to differentiate them tion (dosimetric performance), but also on the accuracy of the dose from overall plan quality. calculation and beam delivery. Therefore, plan quality must incor- In conclusion, the dose distribution delivered to the patient porate an assessment of plan complexity. Plan complexity has been depends not only on the calculated dose distribution but also on discussed in the literature mainly for photon radiotherapy based the robustness and complexity of the treatment plan. Therefore, on conventional linear accelerators, but we believe that some of all these characteristics need to be taken into account when plan these complexity metrics would be valid for other techniques such quality is evaluated. Plan quality should be linked to clinical out- as robotic radiosurgery (CyberKnife) and similar concepts that come and we believe that, to that aim, collecting structured data reduce complexity and increase efficiency could also be applied on dose metrics, robustness and complexity as well as clinical to proton therapy. information from patients is essential. A better understanding of Many commercial TPSs offer the possibility to control the num- these concepts and further implementation of appropriate tools ber of MUs or the treatment time of the plans they produce. Some in commercial TPSs will facilitate the handling of these character- TPSs incorporate methods to restrict complexity during optimiza- istics in clinical practice and be helpful for increasing the overall tion, e.g. the aperture shape controller (ASC) tool in the Eclipse quality of treatment plans in radiotherapy. TPS [94] and the modulation factor (MF) in the TomoTherapy Hi- Art treatment system [95–97], but most TPSs still do not directly Declaration of Competing Interest incorporate advanced complexity metrics [77]. Thus, unnecessary plan complexity is sometimes a consequence of inverse planning The authors declare that they have no known competing finan- and better tools are needed to handle plan complexity [98]. To that cial interests or personal relationships that could have appeared aim, we encourage TPS manufacturers to: to influence the work reported in this paper.  Minimise and better control plan complexity by incorporating complexity metrics into the optimisation algorithms. In our References opinion, a combination of metrics focused on aperture modula- [1] Jensen K, Friborg J, Hansen CR, Samsøe E, Johansen J, Andersen M, et al. The tion and aperture irregularity could be used, but further studies Danish Head and Neck Cancer Group (DAHANCA) 2020 radiotherapy are necessary. guidelines. Radiother Oncol 2020;151:149–51. https://doi.org/10.1016/j.  Include tools for scoring and reporting on different aspects of radonc.2020.07.037. In press. [2] Hansen CR, Johansen J, Samsøe E, Andersen E, Petersen JBB, Jensen K, et al. the complexity of treatment plans to facilitate the handling of Consequences of introducing geometric GTV to CTV margin expansion in plan complexity during the treatment planning process. DAHANCA contouring guidelines for head and neck radiotherapy. Radiother Oncol 2018;126:43–7. https://doi.org/10.1016/j.radonc.2017.09.019. [3] Tol JP, Delaney AR, Dahele M, Slotman BJ, Verbakel WFAR. Evaluation of a In our opinion, implementation of such tools in TPSs will be knowledge-based planning solution for head and neck cancer. Int J Radiat useful for reducing and rationalising the complexity of clinical Oncol Biol Phys 2015;91:612–20. https://doi.org/10.1016/j. plans and their associated uncertainties, as well as for facilitating ijrobp.2014.11.014. [4] Hussein M, Heijmen BJM, Verellen D, Nisbet A. Automation in intensity a better evaluation of the overall quality of radiotherapy treatment modulated radiotherapy treatment planning-a review of recent innovations. plans. Br J Radiol 2018;1092:20180270. https://doi.org/10.1259/bjr.20180270. 30 V. Hernandez, Christian Rønn Hansen, L. Widesott et al. Radiotherapy and Oncology 153 (2020) 26–33 [5] Tol JP, Doornaert P, Witte BI, Dahele M, Slotman BJ, Verbakel WFAR. A [30] Nelms BE, Robinson G, Markham J, Velasco K, Boyd S, Narayan S, et al. longitudinal evaluation of improvements in radiotherapy treatment plan Variation in external beam treatment plan quality: an inter-institutional quality for head and neck cancer patients. Radiother Oncol 2016;119:337–43. study of planners and planning systems. Pract Radiat Oncol 2012;2:296–305. https://doi.org/10.1016/j.radonc.2016.04.011. https://doi.org/10.1016/j.prro.2011.11.012. [6] Xiao Y, Papiez L, Paulus R, Timmerman R, Straube WL, Bosch WR, et al. [31] Ahmed S, Nelms B, Gintz D, Caudell J, Zhang G, Moros EG, et al. A method for a Dosimetric evaluation of heterogeneity corrections for RTOG 0236: priori estimation of best feasible DVH for organs-at-risk: Validation for head stereotactic body radiotherapy of inoperable stage I-II non-small-cell lung and neck VMAT planning. Med Phys 2017;44:5486–97. https://doi.org/ cancer. Int J Radiat Oncol Biol Phys 2009;73:1235–42. https://doi.org/ 10.1002/mp.12500. 10.1016/j.ijrobp.2008.11.019. [32] Alfonso JCL, Herrero MA, Nunez L. A dose-volume histogram based decision- [7] ICRU. ICRU Report 83. International Commission on Radiation Units and support system for dosimetric comparison of radiotherapy treatment plans. Measurements. Prescribing, recording, and reporting photon-beam intensity- Radiat Oncol 2015;10:263. https://doi.org/10.1186/s13014-015-0569-3. modulated radiation therapy (IMRT). ICRU Report 83. J ICRU 2010;10:1–106. [33] Moustakis C, Chan MKH, Kim J, Nilsson J, Bergman A, Bichay TJ, et al. [8] ICRU. ICRU Report 91. Prescribing, recording and reporting of stereotactic Treatment planning for spinal radiosurgery. Strahlentherapie Und Onkol treatments with small photon beams. J ICRU 2014;14:1–160. 2018;194:843–54. https://doi.org/10.1007/s00066-018-1314-2. [9] Andreo P. Dose to water-like media or dose to tissue in MV photons [34] Hardcastle N, Bignell F, Nelms B, Siva S, Kneebone A, Lao L, et al. The challenge radiotherapy treatment planning: still a matter of debate. Phys Med Biol of planning vertebral body SBRT: Optimizing target volume coverage. Med 2014;60:309–37. https://doi.org/10.1088/0031-9155/60/1/309. Dosim 2020;45:302–7. https://doi.org/10.1016/j.meddos.2020.02.005. [10] Emami B, Lyman J, Brown A, Cola L, Goitein M, Munzenrider JE, et al. [35] Radiation Knowledge n.d. https://radiationknowledge.org/. Tolerance of normal tissue to therapeutic irradiation. Int J Radiat Oncol Biol [36] Fusella M, Scaggion A, Pivato N, Rossato MA, Zorz A, Paiusco M. Efficiently Phys 1991;21:109–22. https://doi.org/10.1016/0360-3016(91)90171-y. train and validate a RapidPlan model through APQM scoring. Med Phys [11] Bentzen SM, Constine LS, Deasy JO, Eisbruch A, Jackson A, Marks LB, et al. 2018;45:2611–9. https://doi.org/10.1002/mp.12896. Quantitative analyses of normal tissue effects in the clinic (QUANTEC): an [37] Monti S, Palma G, D’Avino V, Gerardi M, Marvaso G, Ciardo D, et al. Voxel- introduction to the scientific issues. Int J Radiat Oncol Biol Phys 2010;76: based analysis unveils regional dose differences associated with radiation- S3–9. https://doi.org/10.1016/j.ijrobp.2009.09.040. induced morbidity in head and neck cancer patients. Sci Rep 2017;7:1–8. [12] Hansen CR, Bertelsen A, Zukauskaite R, Johnsen L, Bernchou U, Thwaites DI, https://doi.org/10.1038/s41598-017-07586-x. et al. Prediction of radiation-induced mucositis of H&N cancer patients based [38] Faught AM, Yamamoto T, Castillo R, Castillo E, Zhang J, Miften M, et al. on a large patient cohort. Radiother Oncol 2020;147:15–21. https://doi.org/ Evaluating which dose-function metrics are most critical for functional- 10.1016/j.radonc.2020.03.013. guided radiation therapy. Int J Radiat Oncol Biol Phys 2017;99:202–9. https:// [13] Grégoire V. Tumor control probability (TCP) and normal tissue complication doi.org/10.1016/j.ijrobp.2017.03.051. probability (NTCP) in head and neck cancer. Rays 2005;30:105–8. [39] Liang B, Yan H, Tian Y, Chen X, Yan L, Zhou Z, et al. Dosiomics: extracting 3D [14] Sharabiani M, Clementel E, Andratschke N, Hurkmans C. Generalizability spatial features from dose distribution to predict incidence of radiation assessment of head and neck cancer NTCP models based on the TRIPOD pneumonitis. Front Oncol 2019;9:269. https://doi.org/ criteria. Radiother Oncol 2020;146:143–50. https://doi.org/10.1016/j. 10.3389/fonc.2019.00269. radonc.2020.02.013. [40] Lakshminarayanan P, Jiang W, Robertson SP, Cheng Z, Han P, Bowers M, et al. [15] Hansen CR, Friborg J, Jensen K, Samsøe E, Johnsen L, Zukauskaite R, et al. NTCP Radio-morphology: Parametric shape-based features in radiotherapy. Med model validation method for DAHANCA patient selection of protons versus Phys 2019;46:704–13. https://doi.org/10.1002/mp.13323. photons in head and neck cancer radiotherapy. Acta Oncol (Madr) [41] Rossi L, Bijman R, Schillemans W, Aluwini S, Cavedon C, Witte M, et al. 2019;58:1410–5. https://doi.org/10.1080/0284186X.2019.1654129. Texture analysis of 3D dose distributions for predictive modelling of toxicity [16] Dionisi F, Fiorica F, D’Angelo E, Maddalo M, Giacomelli I, Tornari E, et al. rates in radiotherapy. Radiother Oncol 2018;129:548–53. https://doi.org/ Organs at risk’s tolerance and dose limits for head and neck cancer re- 10.1016/j.radonc.2018.07.027. irradiation: A literature review. Oral Oncol 2019;98:35–47. https://doi.org/ [42] Lee SH, Han P, Hales R, Voong KR, Noro K, Sugiyama S, et al. Multi-view 10.1016/j.oraloncology.2019.08.017. radiomics and dosiomics analysis with machine learning for predicting acute- [17] Yaparpalvi R, Garg MK, Shen J, Bodner WR, Mynampati DK, Gafar A, et al. phase weight loss in lung cancer patients treated with radiotherapy. Phys Evaluating which plan quality metrics are appropriate for use in lung SBRT. Br Med Biol 2020. https://doi.org/10.1088/1361-6560/ab8531. In press. J Radiol 2018;91:20170393. https://doi.org/10.1259/bjr.20170393. [43] Zwanenburg A, Vallières M, Abdalah MA, Aerts HJWL, Andrearczyk V, Apte A, [18] Yoon M, Park SY, Shin D, Lee SB, Pyo HR, Kim DY, et al. A new homogeneity et al. The Image Biomarker Standardization Initiative: standardized index based on statistical analysis of the dose–volume histogram. J Appl Clin quantitative radiomics for high-throughput image-based phenotyping. Med Phys 2007;8:9–17. https://doi.org/10.1120/jacmp.v8i2.2390. Radiology 2020;295:328–38. https://doi.org/10.1148/radiol.2020191145. [19] Yan L, Xu Y, Chen X, Xie X, Liang B, Dai J. A new homogeneity index definition [44] Placidi L, Lenkowicz J, Cusumano D, Boldrini L, Dinapoli N, Valentini V. for evaluation of radiotherapy plans. J Appl Clin Med Phys 2019;20:50–6. Stability of dosomics features extraction on grid resolution and algorithm for https://doi.org/10.1002/acm2.12739. radiotherapy dose calculation. Phys Medica 2020;77:30–5. https://doi.org/ [20] Feuvret L, Noël G, Mazeron J-J, Bey P. Conformity index: a review. Int J Radiat 10.1016/j.ejmp.2020.07.022. Oncol Biol Phys 2006;64:333–42. https://doi.org/10.1016/j. [45] Stroom JC, de Boer HCJ, Huizenga H, Visser AG. Inclusion of geometrical ijrobp.2005.09.028. uncertainties in radiotherapy treatment planning by means of coverage [21] Shaw E, Kline R, Gillin M, Souhami L, Hirschfeld A, Dinapoli R, et al. Radiation probability. Int J Radiat Oncol Biol Phys 1999;43:905–19. https://doi.org/ Therapy Oncology Group: radiosurgery quality assurance guidelines. Int J 10.1016/s0360-3016(98)00468-4. Radiat Oncol Biol Phys 1993;27:1231–9. https://doi.org/10.1016/0360-3016 [46] Van Herk M, Remeijer P, Rasch C, Lebesque JV. The probability of correct (93)90548-a. target dosage: dose-population histograms for deriving treatment margins in [22] Paddick I, Lippitz B. A simple dose gradient measurement tool to complement radiotherapy. Int J Radiat Oncol Biol Phys 2000;47:1121–35. https://doi.org/ the conformity index. J Neurosurg 2006;105:194–201. https://doi.org/ 10.1016/s0360-3016(00)00518-6. 10.3171/sup.2006.105.7.194. [47] McKenzie A, van Herk M, Mijnheer B. Margins for geometric uncertainty [23] Reynolds TA, Jensen AR, Bellairs EE, Ozer M. Dose gradient index for around organs at risk in radiotherapy. Radiother Oncol 2002;62:299–307. stereotactic radiosurgery/radiation therapy. Int J Radiat Oncol Biol Phys https://doi.org/10.1016/s0167-8140(02)00015-4. 2020;106:604–11. https://doi.org/10.1016/j.ijrobp.2019.11.408. [48] Stroom JC, Heijmen BJM. Limitations of the planning organ at risk volume [24] Giglioli FR, Garibaldi C, Blanck O, Villaggi E, Russo S, Esposito M, et al. (PRV) concept. Int J Radiat Oncol Biol Phys 2006;66:279–86. https://doi.org/ Dosimetric multicenter planning comparison studies for stereotactic body 10.1016/j.ijrobp.2006.05.009. radiation therapy: methodology and future perspectives. Int J Radiat Oncol [49] Gordon JJ, Siebers JV. Evaluation of dosimetric margins in prostate IMRT Biol Phys 2020;106:403–12. https://doi.org/10.1016/j.ijrobp.2019.10.041. treatment plans. Med Phys 2008;35:569–75. https://doi.org/10.1118/ [25] Akpati H, Kim C, Kim B, Park T, Meek A. Unified dosimetry index (UDI): a 1.2826558. figure of merit for ranking treatment plans. J Appl Clin Med Phys [50] Lomax AJ. Intensity modulated proton therapy and its sensitivity to treatment 2008;9:99–108. https://doi.org/10.1120/jacmp.v9i3.2803. uncertainties 1: The potential effects of calculational uncertainties. Phys Med [26] Ruan D, Shao W, Demarco J, Tenn S, King C, Low D, et al. Evolving treatment Biol 2008;53:1027–42. https://doi.org/10.1088/0031-9155/53/4/014. plan quality criteria from institution-specific experience. Med Phys [51] Lomax AJ. Intensity modulated proton therapy and its sensitivity to 2012;39:2708–12. https://doi.org/10.1118/1.4704497. treatment uncertainties 2: The potential effects of inter-fraction and inter- [27] Jornet N, Carrasco P, Beltrán M, Calvo JF, Escudé L, Hernández V, et al. field motions. Phys Med Biol 2008;53:1043–56. https://doi.org/10.1088/ Multicentre validation of IMRT pre-treatment verification: comparison of in- 0031-9155/53/4/015. house and external audit. Radiother Oncol 2014;112:381–8. https://doi.org/ [52] Unkelbach J, Chan TCY, Bortfeld T. Accounting for range uncertainties in the 10.1016/j.radonc.2014.06.016. optimization of intensity modulated proton therapy. Phys Med Biol [28] Ventura T, do Carmo Lopes M, Ferreira BC, Khouri L. SPIDERplan: A tool to 2007;52:2755–73. https://doi.org/10.1088/0031-9155/52/10/009. support decision-making in radiation therapy treatment plan assessment. [53] Liu W, Frank SJ, Li X, Li Y, Zhu RX, Mohan R. PTV-based IMPT optimization Reports Pract Oncol Radiother 2016;21:508–16. https://doi.org/10.1016/j. incorporating planning risk volumes vs robust optimization. Med Phys rpor.2016.07.002. 2013;40:21709. https://doi.org/10.1118/1.4774363. [29] Mambretti M, Romanò C, Marvaso G, Comi S, Cambria R, Ciardo D, et al. A [54] Zhang X, Rong Y, Morrill S, Fang J, Narayanasamy G, Galhardo E, et al. global Unified Dosimetry Index (gUDI) to evaluate simultaneous integrated Robust optimization in lung treatment plans accounting for geometric boost radiotherapy plans in prostate cancer. Radiother Oncol uncertainty. J Appl Clin Med Phys 2018;19:19–26. https://doi.org/10.1002/ 2018;128:315–20. https://doi.org/10.1016/j.radonc.2018.06.002. acm2.12291. 31 Plan quality in radiotherapy: dose metrics, complexity, and robustness [55] Unkelbach J, Alber M, Bangert M, Bokrantz R, Chan TCY, Deasy JO, et al. Robust [79] Nauta M, Villarreal-Barajas JE, Tambasco M. Fractal analysis for assessing the radiotherapy planning. Phys Med Biol 2018;63:22TR02.. https://doi.org/ level of modulation of IMRT fields. Med Phys 2011;38:5385–93. https://doi. 10.1088/1361-6560/aae659. org/10.1118/1.3633912. [56] Wieser H-P, Karger CP, Wahl N, Bangert M. Impact of Gaussian [80] Younge KC, Roberts D, Janes LA, Anderson C, Moran JM, Matuszak MM. uncertainty assumptions on probabilistic optimization in particle Predicting deliverability of volumetric-modulated arc therapy (VMAT) plans therapy. Phys Med Biol 2020;65:. https://doi.org/10.1088/1361-6560/ using aperture complexity analysis. J Appl Clin Med Phys 2016;17:124–31. ab8d77145007. https://doi.org/10.1120/jacmp.v17i4.6241. [57] Fredriksson A, Bokrantz R. A critical evaluation of worst case optimization [81] Vieillevigne L, Khamphan C, Saez J, Hernandez V. On the need for tuning the methods for robust intensity-modulated proton therapy planning. Med Phys dosimetric leaf gap for stereotactic treatment plans in the Eclipse treatment 2014;41:. https://doi.org/10.1118/1.4883837081701. planning system. J Appl Clin Med Phys 2019;20:68–77. https://doi.org/ [58] Korevaar EW, Habraken SJM, Scandurra D, Kierkels RGJ, Unipan M, Eenink 10.1002/acm2.12656. MGC, et al. Practical robustness evaluation in radiotherapy–A photon and [82] Hubley E, Pierce G. The influence of plan modulation on the interplay effect in proton-proof alternative to PTV-based plan evaluation. Radiother Oncol VMAT liver SBRT treatments. Phys Medica 2017;40:115–21. https://doi.org/ 2019;141:267–74. https://doi.org/10.1016/j.radonc.2019.08.005. 10.1016/j.ejmp.2017.07.025. [59] Yock AD, Mohan R, Flampouri S, Bosch W, Taylor PA, Gladstone D, et al. [83] De Muinck Keizer DM, Kontaxis C, Kerkmeijer LGW, Der Voort Van, Van Zyp Robustness analysis for external beam radiation therapy treatment plans: JRN, Van Den Berg CAT, et al. Dosimetric impact of soft-tissue based describing uncertainty scenarios and reporting their dosimetric intrafraction motion from 3D cine-MR in prostate SBRT. Phys Med Biol consequences. Pract Radiat Oncol 2019;9:200–7. https://doi.org/10.1016/j. 2020;65. https://doi.org/10.1088/1361-6560/ab6241. 025012. prro.2018.12.002. [84] Ezzell GA, Burmeister JW, Dogan N, LoSasso TJ, Mechalakos JG, Mihailidis D, [60] McGowan SE, Albertini F, Thomas SJ, Lomax AJ. Defining robustness et al. IMRT commissioning: Multiple institution planning and dosimetry protocols: a method to include and evaluate robustness in clinical plans. comparisons, a report from AAPM Task Group 119. Med Phys Phys Med Biol 2015;60:2671–84. https://doi.org/10.1088/0031-9155/60/7/ 2009;36:5359–73. https://doi.org/10.1118/1.3238104. 2671. [85] Craft D, Süss P, Bortfeld T. The tradeoff between treatment plan quality and [61] Malyapa R, Lowe M, Bolsi A, Lomax AJ, Weber DC, Albertini F. required number of monitor units in intensity-modulated radiotherapy. Int J Evaluation of robustness to setup and range uncertainties for head Radiat Oncol Biol Phys 2007;67:1596–605. https://doi.org/10.1016/j. and neck patients treated with pencil beam scanning proton therapy. ijrobp.2006.11.034. Int J Radiat Oncol Biol Phys 2016;95:154–62. https://doi.org/10.1016/j. [86] McGarry CK, Agnew CE, Hussein M, Tsang Y, McWilliam A, Hounsell AR, et al. ijrobp.2016.02.016. The role of complexity metrics in a multi-institutional dosimetry audit of [62] van Dijk LV, Van den Bosch L, Aljabar P, Peressutti D, Both S, Steenbakkers VMAT. Br J Radiol 2016;89:20150445. https://doi.org/10.1259/bjr.20150445. Roel JHM, et al. Improving automatic delineation for head and neck organs at [87] Jurado-Bruggeman D, Hernández V, Sáez J, Navarro D, Pino F, Martinez T, et al. risk by Deep Learning Contouring. Radiother Oncol 2020;142:115–23. Multi-centre audit of VMAT planning and pre-treatment verification. https://doi.org/10.1016/j.radonc.2019.09.022. Radiother Oncol 2017;124:302–10. https://doi.org/10.1016/j. [63] LoSasso T, Chui C-S, Ling CC. Physical and dosimetric aspects of a multileaf radonc.2017.05.019. collimation system used in the dynamic mode for implementing intensity [88] Lee MT, Purdie TG, Eccles CL, Sharpe MB, Dawson LA. Comparison of simple modulated radiotherapy. Med Phys 1998;25:1919–27. https://doi.org/ and complex liver intensity modulated radiotherapy. Radiat Oncol 10.1118/1.598381. 2010;5:115. https://doi.org/10.1186/1748-717X-5-115. [64] Das IJ, Ding GX, Ahnesjö A. Small fields: nonequilibrium radiation dosimetry. [89] Matuszak MM, Larsen EW, Fraass BA. Reduction of IMRT beam complexity Med Phys 2008;35:206–15. https://doi.org/10.1118/1.2815356. through the use of beam modulation penalties in the objective function. Med [65] Miften M, Olch A, Mihailidis D, Moran J, Pawlicki T, Molineu A, et al. Tolerance Phys 2007;34:507–20. https://doi.org/10.1118/1.2409749. limits and methodologies for IMRT measurement-based verification QA: [90] Babier A, Boutilier JJ, McNiven AL, Chan TCY. Knowledge-based automated recommendations of AAPM Task Group No. 218. Med Phys 2018;45:e53–83. planning for oropharyngeal cancer. Med Phys 2018;45:2875–83. https://doi. https://doi.org/10.1002/mp.12810. org/10.1002/mp.12930. [66] Johnson NF. Two’s company, three is complexity. Simply Complex. A Clear [91] Hansen CR, Nielsen M, Bertelsen AS, Hazell I, Holtved E, Zukauskaite R, et al. Guid. to Complex. theory. Oneworld Publications; 2009. p. 3. Automatic treatment planning facilitates fast generation of high-quality [67] Du W, Cho SH, Zhang X, Hoffman KE, Kudchadker RJ. Quantification of beam treatment plans for esophageal cancer. Acta Oncol (Madr) complexity in intensity-modulated radiation therapy treatment plans. Med 2017;56:1495–500. https://doi.org/10.1080/0284186X.2017.1349928. Phys 2014;41:. https://doi.org/10.1118/1.4861821021716. [92] Hansen CR, Bertelsen A, Hazell I, Zukauskaite R, Gyldenkerne N, Johansen J, [68] Hernandez V, Saez J, Pasler M, Jurado-Bruggeman D, Jornet N. Comparison of et al. Automatic treatment planning improves the clinical quality of head and complexity metrics for multi-institutional evaluations of treatment plans in neck cancer treatment plans. Clin Transl Radiat Oncol 2016;1:2–8. https:// radiotherapy. Phys Imaging Radiat Oncol 2018;5:37–43. https://doi.org/ doi.org/10.1016/j.ctro.2016.08.001. 10.1016/j.phro.2018.02.002. [93] Crijns W, Defraene G, Van Herck H, Depuydt T, Haustermans K, Maes F, et al. [69] McNiven AL, Sharpe MB, Purdie TG. A new metric for assessing IMRT Online adaptation and verification of VMAT. Med Phys 2015;42:3877–91. modulation complexity and plan deliverability. Med Phys 2010;37:505–15. https://doi.org/10.1118/1.4921615. https://doi.org/10.1118/1.3276775. [94] Scaggion A, Fusella M, Agnello G, Bettinelli A, Pivato N, Roggio A, et al. [70] Younge KC, Matuszak MM, Moran JM, McShan DL, Fraass BA, Roberts DA. Limiting treatment plan complexity by applying a novel commercial tool. J Penalization of aperture complexity in inversely planned volumetric Appl Clin Med Phys 2020;21:27–34. https://doi.org/10.1002/acm2.12908. modulated arc therapy. Med Phys 2012;39:7160–70. https://doi.org/ [95] Ryczkowski A, Piotrowski T, et al. Influence of the modulation factor on the 10.1118/1.4762566. treatment plan quality and execution time in Tomotherapy in head and neck [71] Götstedt J, Karlsson Hauer A, Bäck A. Development and evaluation of cancer: In-phantom study. J Cancer Res Ther 2013;9:618–23.. https://doi.org/ aperture-based complexity metrics using film and EPID measurements of 10.4103/0973-1482.126458. static MLC openings. Med Phys 2015;42:3911–21. https://doi.org/10.1118/ [96] Boyd R, Jeong K, Tomé WA. Determining efficient helical IMRT modulation 1.4921733. factor from the MLC leaf-open time distribution on precision treatment [72] Kairn T, Crowe SB, Kenny J, Knight RT, Trapp JV. Predicting the likelihood of planning system. J Appl Clin Med Phys 2019;20:64–74. https://doi.org/ QA failure using treatment plan accuracy metrics. J Phys Conf Ser 10.1002/acm2.12581. 2014;489:12051. https://doi.org/10.1088/1742-6596/489/1/012051. [97] Santos T, Ventura T, Mateus J, Capela M, Lopes M do C. On the complexity of [73] Crowe SB, Kairn T, Kenny J, Knight RT, Hill B, Langton CM, et al. Treatment helical tomotherapy treatment plans. J Appl Clin Med Phys 2020;21:107–18. plan complexity metrics for predicting IMRT pre-treatment quality assurance https://doi.org/10.1002/acm2.12895. results. Australas Phys Eng Sci Med 2014;37:475–82. https://doi.org/10.1007/ [98] Kamperis E, Kodona C, Hatziioannou K, Giannouzakos V. Complexity in s13246-014-0274-9. Radiation Therapy: It’s Complicated. Int J Radiat Oncol Biol Phys [74] Masi L, Doro R, Favuzza V, Cipressi S, Livi L. Impact of plan parameters on the 2020;106:182–4. https://doi.org/10.1016/j.ijrobp.2019.09.003. dosimetric accuracy of volumetric modulated arc therapy. Med Phys [99] Li JS, Lin T, Chen L, Price Jr RA, Ma C-M. Uncertainties in IMRT dosimetry. Med 2013;40:. https://doi.org/10.1118/1.4810969071718. Phys 2010;37:2491–500. https://doi.org/10.1118/1.3413997. [75] Park JM, Park S-Y, Kim H, Kim JH, Carlson J, Ye S-J. Modulation indices for [100] McGarry CK, Bokrantz R, O’Sullivan JM, Hounsell AR. Advantages and volumetric modulated arc therapy. Phys Med Biol 2014;59:7315–40. https:// limitations of navigation-based multicriteria optimization (MCO) for doi.org/10.1088/0031-9155/59/23/7315. localized prostate cancer IMRT planning. Med Dosim 2014;39:205–11. [76] Park JM, Wu HG, Kim JH, Carlson JNK, Kim K. The effect of MLC speed and https://doi.org/10.1016/j.meddos.2014.02.002. acceleration on the plan delivery accuracy of VMAT. Br J Radiol [101] Kyroudi A, Petersson K, Ghandour S, Pachoud M, Matzinger O, Ozsahin M, 2015;88:20140698. https://doi.org/10.1259/bjr.20140698. et al. Discrepancies between selected Pareto optimal plans and final [77] Chiavassa S, Bessieres I, Edouard M, Mathot M, Moignier A. Complexity deliverable plans in radiotherapy multi-criteria optimization. Radiother metrics for IMRT and VMAT plans: a review of current literature and Oncol 2016;120:346–8. https://doi.org/10.1016/j.radonc.2016.05.018. applications. Br J Radiol 2019;92:20190270. https://doi.org/10.1259/ [102] Schwarz M, Cattaneo GM, Marrazzo L. Geometrical and dosimetrical bjr.20190270. uncertainties in hypofractionated radiotherapy of the lung: A review. Phys [78] Antoine M, Ralite F, Soustiel C, Marsac T, Sargos P, Cugny A, et al. Use of Medica 2017;36:126–39. https://doi.org/10.1016/j.ejmp.2017.02.011. metrics to quantify IMRT and VMAT treatment plan complexity: A systematic [103] Liebl J, Paganetti H, Zhu M, Winey BA. The influence of patient positioning review and perspectives. Phys Medica 2019;64:98–108. https://doi.org/ uncertainties in proton radiotherapy on proton range and dose distributions. 10.1016/j.ejmp.2019.05.024. Med Phys 2014;41:. https://doi.org/10.1118/1.4892601091711. 32 V. Hernandez, Christian Rønn Hansen, L. Widesott et al. Radiotherapy and Oncology 153 (2020) 26–33 [104] Nassef M, Simon A, Cazoulat G, Duménil A, Blay C, Lafond C, et al. multiple dosimetric parameters, overall scoring index and clinical scoring. Quantification of dose uncertainties in cumulated dose estimation Phys Medica 2018;56:25–33. https://doi.org/10.1016/j.ejmp.2018.11.012. compared to planned dose in prostate IMRT. Radiother Oncol [107] Villaggi E, Hernandez V, Fusella M, Moretti E, Russo S, Vaccara EML, et al. Plan 2016;119:129–36. https://doi.org/10.1016/j.radonc.2016.03.007. quality improvement by DVH sharing and planner’s experience: Results of a [105] Blanck O, Wang L, Baus W, Grimm J, Lacornerie T, Nilsson J, et al. Inverse SBRT multicentric planning study on prostate. Phys Medica 2019;62:73–82. treatment planning for spinal robotic radiosurgery: an international multi- https://doi.org/10.1016/j.ejmp.2019.05.003. institutional benchmark trial. J Appl Clin Med Phys 2016;17:313–30. https:// [108] Russo S, Esposito M, Hernandez V, Saez J, Rossi F, Paoletti L, et al. Does deep doi.org/10.1120/jacmp.v17i3.6151. inspiration breath hold reduce plan complexity? Multicentric experience of [106] Masi L, Zani M, Doro R, Calusi S, Di Cataldo V, Bonucci I, et al. CyberKnife left breast cancer radiotherapy with volumetric modulated arc therapy. Phys MLC-based treatment planning for abdominal and pelvic SBRT: Analysis of Medica 2019;59:79–85. https://doi.org/10.1016/j.ejmp.2019.02.018. 33