Radiotherapy and Oncology 173 (2022) 254–261 Contents lists available at ScienceDirect Radiotherapy and Oncology journal homepage: www.thegreenjournal.com Original Article Plan quality assessment in clinical practice: Results of the 2020 ESTRO survey on plan complexity and robustness Laura Patricia Kaplan a,b,1,⇑, Lorenzo Placidi c,1,⇑, Anna Bäck d,e, Richard Canters f, Mohammad Hussein g, Ana Vaniqui f, Marco Fusella h, Tomasz Piotrowski i, Victor Hernandez j, Nuria Jornet k, Christian Rønn Hansen l,m,n, Lamberto Widesott o a Department of Oncology, Aarhus University Hospital; b Department of Clinical Medicine, Aarhus University, Denmark; c Fondazione Policlinico Universitario ‘‘A. Gemelli’’ IRCCS, Roma, Italy; d Department of Therapeutic Radiation Physics, Sahlgrenska University Hospital, Gothenburg; e Department of Medical Radiation Sciences, Sahlgrenska Academy, University of Gothenburg, Sweden; f Department of Radiation Oncology (Maastro), GROW School for Oncology, Maastricht University Medical Centre+, the Netherlands; g Metrology for Med Phys Centre, National Physical Laboratory, Teddington, United Kingdom; h Department of Med Phys, Veneto Institute of Oncology - IOV IRCCS, Padua, Italy; i Department of Electroradiology, Poznan University of Medical Sciences and Department of Med Phys, Greater Poland Cancer Centre, Poznan, Poland; j Department of Med Phys, Hospital Sant Joan de Reus, IISPV, Spain; k Servei de Radiofísica i Radioprotecció, Hospital de la Santa Creu i Sant Pau, Barcelona, Spain; l Laboratory of Radiation Physics, Department of Oncology, Odense University Hospital; m Danish Centre for Particle Therapy, Aarhus University Hospital; n Department of Clinical Research, University of Southern Denmark, Odense, Denmark; o Centro di protonterapia, APSS, Trento, Italy a r t i c l e i n f o a b s t r a c t Article history: Purpose: Plan complexity and robustness are two essential aspects of treatment plan quality but there is Received 17 February 2022 a great variability in their management in clinical practice. This study reports the results of the 2020 Received in revised form 24 May 2022 ESTRO survey on plan complexity and robustness to identify needs and guide future discussions and con- Accepted 7 June 2022 sensus. Available online 14 June 2022 Methods: A survey was distributed online to ESTRO members. Plan complexity was defined as the mod- ulation of machine parameters and increased uncertainty in dose calculation and delivery. Robustness Keywords: was defined as a dose distribution’s sensitivity towards errors stemming from treatment uncertainties, International survey Plan quality patient setup, or anatomical changes. Plan complexity Results: A total of 126 radiotherapy centres from 33 countries participated, 95 of them (75%) from Europe Plan robustness and Central Asia. The majority controlled and evaluated plan complexity using monitor units (56 centres) and aperture shapes (38 centres). To control robustness, 98 (97% of question responses) photon and 5 (50%) proton centres used PTV margins for plan optimization while 75 (94%) and 5 (50%), respectively, used margins for plan evaluation. Seventeen (21%) photon and 8 (80%) proton centres used robust opti- misation, while 10 (13%) and 8 (80%), respectively, used robust evaluation. Primary uncertainties consid- ered were patient setup (photons and protons) and range calculation uncertainties (protons). Participants expressed the need for improved commercial tools to control and evaluate plan complexity and robust- ness. Conclusion: Clinical implementation of methods to control and evaluate plan complexity and robustness is very heterogeneous. Better tools are needed to manage complexity and robustness in treatment plan- ning systems. International guidelines may promote harmonization. Ó 2022 The Authors. Published by Elsevier B.V. Radiotherapy and Oncology 173 (2022) 254–261 This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/). Radiotherapy (RT) treatment plan quality assessment is a broad Radiotherapy dose distributions can be shaped to be highly con- and complex topic. Plan complexity and robustness are two formal to a target volume using techniques such as intensity mod- aspects within plan quality in which there currently is particular ulated radiotherapy (IMRT), volumetric modulated arc therapy interest and controversy within the scientific community. (VMAT), or intensity modulated proton therapy (IMPT). Increased sculpting of the 3D dose distribution often comes hand in hand ⇑ Corresponding authors at: Department of Oncology and Palliative Units, with increased modulation of many machine parameters such as Zealand University Hospital - Næstved Hospital, Rådmandsengen 5, 4700 Næstved, output rates, gantry speed, or field shapes. This increased modula- Denmark. Fondazione Policlinico Universitario ‘‘A. Gemelli”, IRCCS, UOSD Fisica tion, termed plan complexity, may lead to increased uncertainty in Medica e Radioprotezione, Ala F, piano 2, Largo Agostino Gemelli 8, 00168, Roma. dose calculation and treatment delivery [1–9]. Inaccurate calcula- E-mail addresses: lakap@regionsjaelland.dk (L.P. Kaplan), lorenzo.placidi@poli- tion and delivery of highly complex plans may negatively impact clinicogemelli.it (L. Placidi). 1 Contributed equally to this manuscript. clinical outcomes. Further, they can hinder successful clinical trials https://doi.org/10.1016/j.radonc.2022.06.005 0167-8140/Ó 2022 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/). Laura Patricia Kaplan, L. Placidi, A. Bäck et al. Radiotherapy and Oncology 173 (2022) 254–261 and obstruct detection of dose–response effects in big data analy- entire group with a focus on clarity and completeness of ses. These differences between planned and delivered dose can be multiple-choice answers. The questionnaire was implemented in observed in dosimetry audits [10–12]. an online form and trial runs were performed at some of the work- Several metrics have been proposed to quantify and investigate ing group members’ centres to verify the questions were under- plan complexity [1–8]. Recent reviews give an overview of these stood and the survey was implemented as intended. The final metrics and published studies on their correlation with plan qual- web-based survey (see Supplementary Figure S1 and file ity assurance results [13–15]. Some complexity metrics provide ‘‘Responses.xlsx”) was distributed and promoted via ESTRO mail- similar information and can be considered equivalent. However, ing lists and national medical physics associations between Febru- metrics that focus on different plan parameters yield different ary and May of 2020. results and there is no clear consensus as to which should be used Respondents were asked to provide only one response per cen- [16]. tre. In cases where several responses per centre were recorded, the At the same time, highly conformal treatment plans can come at most complete or (if both were equally complete) the latest the cost of inferior robustness towards treatment errors such as response was chosen. Responding centres were included in the those caused by shifts in patient setup, anatomical changes, or analysis if they had answered at least one question. Centres were range calculation uncertainties (for protons). A lack of robustness excluded if any answers clearly were not intended to answer the towards such errors might in the worst case lead to insufficient tar- posed question (e.g., ‘‘ABC” as an answer to an open-ended ques- get coverage or severe side-effects due to over-dosing of critical tion). Single answers were disregarded if there were obvious healthy tissue. incongruencies (e.g., respondent answered ‘‘No” to performing pro- Until a few years ago, the only universally used method to opti- ton treatment planning but was still, due to errors in the survey mize and evaluate plan robustness was defining margins around software, shown questions concerning proton planning). the clinical target volume (CTV) and organs at risk (OAR) to obtain Cross-analyses were performed to investigate whether there the planning target volume (PTV) and planning organ at risk vol- was a correlation between how centres regarded plan complexity ume (PRV), respectively. Several formulas have been proposed in and robustness, respectively. the literature for the definition of PTV [17,18] and PRV [19,20] mar- gins. The concept of safety margins has inherent limitations, how- Structure of the survey ever; the largest being the assumption that the spatial dose distribution does not change shape under the influence of errors The survey consisted of four sections (see supplementary (static dose cloud approximation). It has therefore been high- Fig. S1). The first included four general questions about the centre’s lighted that margins are inadequate for use in particle therapy, location and the software and hardware used for treatment plan- and possibly also in highly modulated photon treatments [20–23]. ning and delivery. Robust optimization (RO) has been suggested as a method to The second covered treatment plan complexity. Complexity was potentially overcome the limitations of safety margins and thus defined as increased modulation of many machine parameters improve reliable CTV coverage and OAR sparing both for photon resulting in increased demand on the accuracy of dose calculation and proton treatments [21,24–26]. RO directly calculates the dose and treatment delivery. Centres were asked for which types of changes induced by simulated error scenarios, thus taking both the treatments they considered plan complexity to be an issue (if patient-specific anatomy and dose distribution characteristics any), and how they controlled and evaluated it. No distinction (field directions, penumbra, dose gradient. . .) into account. Robust was made between photon and proton treatments in this section. evaluation (RE) follows the same concept as RO: the optimized The third section was focused on plan robustness. Robustness dose is recalculated and evaluated in simulated error scenarios. was defined as the dose distribution’s sensitivity towards errors Importantly, the terms ‘complexity’, ‘robust optimization’, and stemming from treatment uncertainties, including patient setup, ‘robust evaluation’ cover several different methodologies and met- anatomical changes, and proton range uncertainties. The survey rics, and there is no consensus yet on which to implement or how aimed at answering two overarching questions about plan robust- to use them [23,27–30]. Given the potential of these new tools and ness: (1) which methods and metrics do centres use to control and their increasing availability in treatment planning systems, the sci- evaluate plan robustness? And (2) how do the two main entific community needs to discuss which methods are most approaches of controlling and evaluating robustness, safety mar- appropriate. gins and RO/RE, compare in terms of uncertainty types considered During the third edition of the ‘European Society for Radiother and sizes of margins/simulated patient shifts? Oncol Physics Workshop: Science in Development’ in October Finally, responding centres were asked which other methods 2019, a dedicated track was focused on the topic ‘Plan quality and metrics they used to assess plan quality (with emphasis on assessment: dose distribution and robustness metrics’. The work- dosimetric quality), and whether they had any requests for future shop group identified treatment plan complexity and robustness implementations in commercial treatment planning systems. as areas of special interest and controversy. The group’s first aim was to map current clinical practice concerning plan complexity Results and robustness assessment both during the plan optimization and evaluation phases. The aim of the 2020 ESTRO online survey General on plan quality assessment was to provide this overview. In this paper, we present and analyse the results of this survey. A total of 126 radiotherapy centres (of 140 recorded responses) These results can be used as the first step towards standardizing were included in the analysis. Of these, 95 (75 %) were from Europe the clinical optimization and evaluation of plan complexity and and Central Asia. Details on the characteristics of participating cen- robustness. tres are provided in supplementary Tables S1-S5. The number of responses for each separate question varied between 4 and 126. A spreadsheet detailing all questions, responses, and reasons for Methods excluding answers is provided in the supplementary material (‘‘Responses.xlsx”). Percentages stated in the following are in rela- Development of survey questions began in October 2019 at the tion to the number of responses given to the respective question. workshop. After the workshop, questions were reviewed by the For evaluation of a plan’s dosimetric quality (excluding complexity 255 Plan quality assessment - 2020 ESTRO survey and robustness), visual inspection and use of DVH curves and met- Complexity rics were the methods used most widely (66, 69, and 64 of 76 Plans in the head-and-neck region were considered most chal- responses; Fig. 1, top). lenging with respect to plan complexity by most responding cen- tres (72 of 95). Machine output (monitor units per Gray) and Fig. 1. Plan complexity. Fig. 1: Responses to the questions ‘‘Which methods (other than those covered in previous questions) do you in your clinical practice use to evaluate a plan’s dosimetric quality?” (a), ‘‘How is complexity controlled during optimization?” (b), ‘‘How is complexity evaluated after optimization?” (c), and ‘‘What do you do with the information gained on complexity?” (d). DQ: Dosimetric quality, PC: Plan complexity, PSV: patient specific verification, PQ: Plan quality, Conf: conformity, Hom.: Homogeneity. 256 Laura Patricia Kaplan, L. Placidi, A. Bäck et al. Radiotherapy and Oncology 173 (2022) 254–261 aperture shapes were the methods mentioned most often to con- using RO/RE (Fig. 3). For example, 48 of 85 (56 %) and 27 of 60 trol and evaluate complexity (Fig. 1). Additionally, patient- (45 %) respondents reported including couch positioning or rota- specific verification measurements were reported equally often tion uncertainties in their PTV and PRV margins, while 4 of 17 as an evaluation method. (24 %) and 1 of 10 (10 %) reported including these in their RO A large proportion of centres reported that they tried reducing and RE, respectively. the complexity of highly complex plans if this was possible while Reported PRV margin sizes were overall smaller than PTV (Sup- keeping dosimetric plan quality constant (67 of 86). Ten centres plementary Figure S2). Two centres reported using different shift would even accept lower dosimetric quality in exchange for lower sizes for RE than for RO. Reported RO shift sizes tended to be smal- complexity. ler than PTV sizes in thoracic and pelvic treatment sites (Supple- Twenty-five centres reported that they did not take complexity mentary Figure S3). into account as part of their clinical evaluation routine at all, due to: Proton centres - lack of clear guidance on how to evaluate complexity (14 Of the 10 responding proton centres, more reported using RO answers) and RE than PTV or PRV (Table 1). Four centres reported using both - because they had no tools to do so (11 answers) PTV and PRV margins for plan optimization, while one centre - because there was no consensus on which metrics to use (8 answered ‘‘Yes” only to using PTV and another only to using PRV answers) margins. Four centres reported using both RO/RE and either PTV - because it was considered too time-consuming (5 answers). or PRV (two using both PTV and PRV, one using only PTV, and one using only PRV). Most centres reported including setup uncertainties in their Robustness margin or shift considerations (Figs. 2 and 3). Range uncertainties Table 1 shows an overview of reported methods used to control were included by all centres using RO and seven of eight using RE, and evaluate plan robustness in photon and proton planning. while only one centre using PTV and another using PRV considered Of 105 centres responding to at least one question in the range (both centres also reported using RO and RE). robustness section, 102 performed photon treatment planning All centres except two reported using the same shift sizes for RE and 10 performed proton treatment planning. Seven clinics per- as for RO (one used smaller shifts for RE and one did not use RO). formed both photon and proton planning. Cross-analysis Photon centres Of photon centres that considered complexity, a larger fraction Ninety-eight photon centres reported using PTV margins for performed RO for targets (19 %), RO for OARs (13 %) and RE (13 %) plan optimization, while only 64 used PRV. Substantially more cen- compared to those that did not consider complexity. tres used safety margins for plan optimization and evaluation than A higher percentage of proton centres included complexity RO and RE (Table 1). Of those photon centres using RO/RE, all but (91 %) in their clinical planning routines compared to photon cen- one reported using both RO/RE and PTV/PRV. tres. Of those proton centres who took plan complexity into PTV margin sizes were mostly determined based on historical account, most used RO for targets (70 %), RO for OARs (80 %) and choices (32 of 85, Fig. 2) or using a formula such as the one by RE (70 %). (Supplementary Figures S4 and S5). van Herk et al. [18] (26 of 85). PRV margin sizes were more often determined based on a historical choice than by a formula (35 vs Suggestions for improvements of commercial complexity and 4, 60 respondents). RO scenarios were calculated most often by robustness tools assigning fraction-specific uncertainty scenarios and summing these (7 of 17 responses). When asked whether they needed better commercial tools for The majority of centres reported including possible setup errors plan complexity management, 61 of 109 respondents requested in their considerations for margins or robustness shifts (Fig. 3). The improved tools for controlling complexity during plan optimiza- distribution of other possible error sources included in margin/ tion, while 70 required tools to evaluate complexity after optimiza- shift considerations varied between centres using margins vs those tion. Twenty-four centres did not see the need for improvement in currently available tools. Open-answer suggestions to improve RO and RE tools mainly Table 1 aimed at making robustness scenarios less simplistic and/or con- Overview of participants who report using margins or robust optimization/evaluation to control and evaluate treatment plan robustness. Figures in parentheses show the servative. Suggestions included: total number of centres responding to each question. RO: robust optimization, RE: robust evaluation. a) Rather than using conservative ‘‘worst-case” scenarios, RO/ Photons Protons RE scenarios should be based on error distributions (5 comments). Optimization Use PTV margin 98 (101) 5 (10) b) Various possible error sources such as couch pitch, roll, or Use RO for target 17 (82) 8 (10) rotation and anatomical variations should be included in Of those using RO, also use PTV 16 (17) 3 (8) tools (2 comments). Use PRV margin 64 (88) 5 (10) c) Better tools for visualization and reporting (1 comment). Use RO for OAR 10 (17) 8 (8) d) Tools for RO/RE should be faster (2 comments). Of those using RO for OAR, also use PRV 7 (10) 3 (8) Evaluation Use PTV/PRV 75 (80) 1 (7) Discussion Use RE 10 (78) 8 (10) Answered ‘‘No” but commented 4 (78) The results of this international survey paint a picture of how RT ‘‘Occasionally” or similar plan complexity and robustness are considered in clinics (as of Of those using RE, also use PTV/PRV 10 (10) 1 (8) 2020). 257 Plan quality assessment - 2020 ESTRO survey Fig. 2. How are safety margins and RO calculated?. Fig. 2: Methods used by photon (A) and proton (B) centres to calculate safety margins and robust optimization shifts. Answer options for PTV/PRV were ‘‘Based on a historical approach”, ‘‘Translate setup errors directly into margin”, ‘‘According to the formula by van Herk et al. or others”, ‘‘We use a probabilistic approach”, and ‘‘Other” (with the option to specify). ‘‘Other” responses included e.g. ‘‘A combination of several approaches”, ‘‘Depends on the diagnosis”, ‘‘Depends on the specific machine uncertainty”, and ‘‘Depends on the physician”. We saw that different methods, often more than one, were often given for why plan complexity was not considered was employed in clinical practice to evaluate a plan’s dosimetric qual- because there was a lack of clear guidance and consensus on ity. Some of the evaluations were performed by inspecting the spa- how to evaluate complexity. The different approaches used by par- tial 3D dose distribution visually. Evaluating the DVH curve and ticipants to control and evaluate complexity also highlight the specific DVH metrics remained the dominant methods to assess absence of both international consensus and of any known toler- dosimetric quality. ance levels for the different metrics. A method to translate com- Additional methods have recently been proposed which could plexity into plan quality and deliverability could facilitate the use aid in better quantifying and objectively assessing a plan’s dosi- of plan complexity in clinical practice. metric quality. Tools such as a total plan quality index (PQI), which Even though we saw different approaches among centres on collapses several indices into an overall plan score [31–35], the use how to deal with plan complexity, there was nevertheless a strong of machine and deep learning [36–38], and new kinds of dose dis- and shared demand for access to commercial tools able to control tribution metrics [39–43] can provide quantification of parameters and evaluate complexity during and after optimization. Several which are currently mainly evaluated subjectively. We believe other authors [3,44] suggest in particular incorporating complexity using such tools will facilitate consensus and standardization on metrics into optimisation algorithms. Quantification of plan com- how to define a treatment plan’s quality. plexity using dedicated metrics can prevent unacceptably high An interesting result of this survey was that 26 of the respond- dosimetric uncertainties - which could lead to substantial differ- ing centres (21 %) did not consider plan complexity at all in their ences between planned and delivered dose if undetected. For this daily practice. Only five of these centres stated that the reason reason, the RATING guidelines for planning studies [45] explicitly for this was lack of time and resources. Instead, the reason most recommend quantitative reporting on the complexity of treatment 258 Laura Patricia Kaplan, L. Placidi, A. Bäck et al. Radiotherapy and Oncology 173 (2022) 254–261 Fig. 3. Which uncertainties are included in margin/shift considerations?. Fig. 3: Uncertainties included in the margin or robust optimization/evaluation considerations by participating photon (A) and proton (B) centres. Each row corresponds to one responding centre. Numbers do not correspond to the same centre in each of the four plots. Abbreviations: anat.: anatomical, ch.: changes, sim.: simulated. plans. Better commercial tools for controlling complexity during was no data on the subject and they considered all modulated optimisation, in particular for automated planning, are highly spot-scanning proton plans equally complex. desirable. For evaluation, vendors are encouraged to enable calcu- Concerning plan robustness, we saw large variations between lation and reporting of complexity metrics. responding centres in how safety margin and RO/RE shift sizes In the questions concerning plan complexity, no distinction was were chosen, which approach was used for RO, and which types made between photon and proton centres. Traditionally, plan com- of uncertainties were considered in margin or RO/RE shift sizes. plexity has been described mainly in the context of photon treat- This variability highlights the need for broad international consen- ments with conventional linear accelerators, but some sus and clear guidelines on how plan robustness should be opti- complexity metrics may be valid for other techniques such as mized, evaluated, and reported. robotic radiosurgery [46]. We believe similar concepts that reduce Some respondents admitted to not knowing how margin sizes complexity and increase efficiency could also be applied to proton were specified, which uncertainties were considered, or the actual therapy. Two of three responding centres who performed only pro- margin sizes, as these decisions were made solely by the treating ton planning commented that they evaluated complexity in terms physicians. This is troubling, since detailed knowledge of the of spot weighting and spot positions, while one answered that they underlying physics is essential to understand the effect of uncer- did not control or evaluate complexity for proton plans since there tainties and errors on the delivered dose. As is recommended in 259 Plan quality assessment - 2020 ESTRO survey the International Commission on Radiation Units and Measure- reporting results. Commercial tools must facilitate the inclusion ments Report 62 [47], medical physicists should always be of various types and sources of errors. involved in decisions of this nature alongside radiation oncologists. A limitation of this study lies in the geographical distribution of A substantial proportion (21 %/13 %) of photon centres reported the participants: most respondents were from European countries, using RO or RE, some routinely and some for select cases only. possibly giving rise to bias. Moreover, although a fair number of Most of these also used margins for optimization and evaluation centres responded to the survey, this represents a small proportion of plans, though. This suggests that the photon community is start- of operating radiotherapy clinics and may not fully capture pat- ing to embrace the new paradigm of RO and RE but has not yet terns of clinical practice at a global level. In the future, it would adopted it to the same degree as the proton community has. be desirable to include more centres from a larger number of coun- Most centres using RO and RE stated that they used the same tries and continents in such a survey. This includes especially shift sizes for both. It has previously been suggested that using countries with a higher density of particle centres, since these cen- the same error distributions to optimize and evaluate treatment tres have a long tradition of using robust optimization and evalua- plans may introduce an estimation bias [48]. Evaluation with dif- tion methods. ferent error distributions than those used for optimization is there- This working group aims to support work towards broad con- fore recommended. sensus regarding which metrics and methods are used to control Moreover, many centres reported the same shift sizes for simu- and assess plan complexity and robustness. The results obtained lating systematic and random errors. We believe this to be due to a in this survey highlight trends in how complexity and robustness misunderstanding of the question. are currently controlled and evaluated. Further, they show that Caution is recommended when directly comparing RO/RE shift there is a need for both practical education and broader discussion sizes to margin sizes. It is difficult to translate the PTV concept, a concerning these topics, as well as for development of commercial margin providing 95 % of a prescribed dose to the CTV in 90 % of tools to facilitate their assessment in clinical practice. the treated population, to the RO framework. Most currently avail- In the future, it will be important to evaluate the impact of such able commercial RO solutions provide only uncertainty scenarios new tools through carefully designed studies [44]. Shared consen- based on setup and range errors (anatomical variations have only sus on metrics will be essential for the performance and inter- been implemented by one of the major vendors to date). PTV mar- pretability of such studies. International guidelines on how to gins, on the other hand, directly incorporate the known distribu- manage plan complexity and robustness in the different steps of tions of errors from several other sources (e.g. contouring or the radiotherapy workflow are strongly desired and, to that aim, machine uncertainties). Attempts at a PTV-to-RO translation have further efforts of the community are warranted. been made by e.g. Korevaar et al. [27] or Perkó et al. [49]. The abil- ity to translate the underlying goal of the PTV to RO will be impor- Funding declarations tant in future studies to connect ‘‘old” PTV-based outcomes to those of RO-based plans. LPK was supported by DCCC Radiotherapy - The Danish In the transition from PRV to RO/RE for organs at risk, two cru- National Research Centre for Radiotherapy, Danish Cancer Society cial points must be taken into consideration: 1) To date, in com- (Grant No. R191-A11526) and Danish Comprehensive Cancer Cen- mercial treatment planning systems the target and OARs are tre, and the Danish Cancer Society (Grant No R167-A11003). robustly optimized using the same parameters. This is in contrast to the tradition of some centres to use smaller margins for organs at risk than the target [50], which is also reflected in our survey Declaration of Competing Interest results. 2) Toxicity data and NTCP models are based on the nominal DVH of organs at risk. Requiring the same (or even lower) OAR The authors declare that they have no known competing finan- dose limits than those published by QUANTEC in all uncertainty cial interests or personal relationships that could have appeared scenarios may be too restrictive. Research into this issue is needed to influence the work reported in this paper. to translate OAR dose constraints to the RO/RE framework [26]. Respondents pinpointed the need to define more realistic and Appendix A. Supplementary data versatile uncertainty scenarios for RO and RE including, for instance, errors caused by anatomical variations or mechanical Supplementary data to this article can be found online at and contouring uncertainties. Further, the need for faster tools https://doi.org/10.1016/j.radonc.2022.06.005. was highlighted to make the requested precision compatible with clinical workflows. We strongly believe improvement in these areas will lead to wider adoption of the RO/RE framework. References The survey showed a tendency for photon centres that consid- [1] McNiven AL, Sharpe MB, Purdie TG. A new metric for assessing IMRT ered plan complexity important to more often use RO and/or RE. modulation complexity and plan deliverability. Med Phys 2010;37:505–15. However, due to the small number of responses, it is difficult to [2] Nauta M, Villarreal-Barajas JE, Tambasco M. Fractal analysis for assessing the draw statistically rigorous conclusions on this question. In the con- level of modulation of IMRT fields. Med Phys 2011;38:5385–93. [3] Younge KC, Matuszak MM, Moran JM, McShan DL, Fraass BA, Roberts DA. text of another large survey, it was hypothesized that centres with Penalization of aperture complexity in inversely planned volumetric more staffing resources were more likely to implement ‘‘non- modulated arc therapy. Med Phys 2012;39:7160–70. standard” techniques (such as complexity assessment and RO/RE) [4] Masi L, Doro R, Favuzza V, Cipressi S, Livi L. Impact of plan parameters on the dosimetric accuracy of volumetric modulated arc therapy. Med Phys 2013;40. [51]. We did not include a question concerning centre size and [5] Park JM, Park S-Y, Kim H, Kim JH, Carlson J, Ye S-J. Modulation indices for resources in this survey, and were unable to gather this informa- volumetric modulated arc therapy. Phys Med Biol 2014;59:7315–40. tion subsequently. [6] Du W, Cho SH, Zhang X, Hoffman KE, Kudchadker RJ. Quantification of beam complexity in intensity-modulated radiation therapy treatment plans. Med Based on the results of this study, we recommend that centres Phys 2014;41. moving from using margins to robust optimization and robust [7] Götstedt J, Karlsson Hauer A, Bäck A. Development and evaluation of aperture- evaluation carefully consider how to include the various types of based complexity metrics using film and EPID measurements of static MLC uncertainties in the new framework to maximize plan robustness openings. Med Phys 2015;42:3911–21. [8] Crowe SB, Kairn T, Kenny J, Knight RT, Hill B, Langton CM, et al. Treatment plan towards relevant uncertainties and improve transparency in complexity metrics for predicting IMRT pre-treatment quality assurance results. Australas Phys Eng Sci Med 2014;37:475–82. 260 Laura Patricia Kaplan, L. Placidi, A. Bäck et al. Radiotherapy and Oncology 173 (2022) 254–261 [9] Li JS, Lin T, Chen L, Price RA, Ma C-M. Uncertainties in IMRT dosimetry. Med [30] Malyapa R, Lowe M, Bolsi A, Lomax AJ, Weber DC, Albertini F. Evaluation of Phys 2010;37:2491–500. robustness to setup and range uncertainties for head and neck patients treated [10] Kouloulias VE, Poortmans PM, Bernier J, Horiot J-C, Johansson K-A, Davis B, with pencil beam scanning proton therapy. Int J Radiat Oncol Biol Phys et al. The Quality Assurance programme of the Radiotherapy Group of the 2016;95:154–62. European Organization for Research and Treatment of Cancer (EORTC): a [31] Jornet N, Carrasco P, Beltrán M, Calvo JF, Escudé L, Hernández V, et al. critical appraisal of 20 years of continuous efforts. Eur J Cancer Multicentre validation of IMRT pre-treatment verification: Comparison of in- 2003;39:430–7. house and external audit. Radiother Oncol 2014;112:381–8. [11] Ibbott GS, Followill DS, Molineu HA, Lowenstein JR, Alvarez PE, Roll JE. [32] Ventura T, Lopes MdC, Ferreira BC, Khouri L. SPIDERplan: a tool to support Challenges in credentialing institutions and participants in advanced decision-making in radiation therapy treatment plan assessment. Rep Pract technology multi-institutional clinical trials. Int J Radiat Oncol Biol Phys Oncol Radiother 2016;21:508–16. 2008;71:S71–5. [33] Akpati H, Kim ChangSeon, Kim B, Park T, Meek A. Unified dosimetry index [12] Budiharto T, Musat E, Poortmans P, Hurkmans C, Monti A, Bar-Deroma R, et al. (UDI): a figure of merit for ranking treatment plans. J Appl Clin Med Phys Profile of European radiotherapy departments contributing to the EORTC 2008;9:99–108. Radiation Oncology Group (ROG) in the 21st century. Radiother Oncol [34] Ruan D, Shao W, DeMarco J, Tenn S, King C, Low D, et al. Evolving treatment 2008;88:403–10. plan quality criteria from institution-specific experience. Med Phys [13] Chiavassa S, Bessieres I, Edouard M, Mathot M, Moignier A. Complexity metrics 2012;39:2708–12. for IMRT and VMAT plans: a review of current literature and applications. Br J [35] Mambretti M, Romanò C, Marvaso G, Comi S, Cambria R, Ciardo D, et al. A Radiol 2019;92:20190270. global unified dosimetry index (gUDI) to evaluate simultaneous integrated [14] Kamperis E, Kodona C, Hatziioannou K, Giannouzakos V. Complexity in boost radiotherapy plans in prostate cancer. Radiother Oncol radiation therapy: it’s complicated. Int J Radiat Oncol Biol Phys 2018;128:315–20. 2020;106:182–4. [36] Kalet AM, Luk SM, Phillips MH. Radiation therapy quality assurance tasks and [15] Antoine M, Ralite F, Soustiel C, Marsac T, Sargos P, Cugny A, et al. Use of metrics tools: the many roles of machine learning. Med Phys 2020;47:e168–77. to quantify IMRT and VMAT treatment plan complexity: a systematic review [37] Fan J, Xing L, Ma M, Hu W, Yang Y. Verification of the machine delivery and perspectives. Phys Med 2019;64:98–108. parameters of a treatment plan via deep learning. Phys Med Biol 2020;65. [16] Hernandez V, Saez J, Pasler M, Jurado-Bruggeman D, Jornet N. Comparison of 195007. complexity metrics for multi-institutional evaluations of treatment plans in [38] Pillai M, Adapa K, Das SK, Mazur L, Dooley J, Marks LB, et al. Using artificial radiotherapy. Phys Imaging Radiat Oncol 2018;5:37–43. intelligence to improve the quality and safety of radiation therapy. J Am Coll [17] Stroom JC, de Boer HCJ, Huizenga H, Visser AG. Inclusion of geometrical Radiol 2019;16:1267–72. uncertainties in radiotherapy treatment planning by means of coverage [39] Rossi L, Bijman R, Schillemans W, Aluwini S, Cavedon C, Witte M, et al. Texture probability. Int J Radiat Oncol Biol Phys 1999;43:905–19. analysis of 3D dose distributions for predictive modelling of toxicity rates in [18] van Herk M, Remeijer P, Rasch C, Lebesque JV. The probability of correct target radiotherapy. Radiother Oncol 2018;129:548–53. dosage: dose-population histograms for deriving treatment margins in [40] Placidi L, Lenkowicz J, Cusumano D, Boldrini L, Dinapoli N, Valentini V. Stability radiotherapy. Int J Radiat Oncol Biol Phys 2000;47:1121–35. of dosomics features extraction on grid resolution and algorithm for [19] McKenzie A, van Herk M, Mijnheer B. Margins for geometric uncertainty radiotherapy dose calculation. Phys Med 2020;77:30–5. around organs at risk in radiotherapy. Radiother Oncol 2002;62:299–307. [41] Vaniqui A, Canters R, Vaassen F, Hazelaar C, Lubken I, Kremer K, et al. [20] Stroom JC, Heijmen BJM. Limitations of the planning organ at risk volume Treatment plan quality assessment for radiotherapy of rectal cancer patients (PRV) concept. Int J Radiat Oncol Biol Phys 2006;66:279–86. using prediction of organ-at-risk dose metrics. Phys Imaging Radiat Oncol [21] Liu W, Frank SJ, Li X, Li Y, Zhu RX, Mohan R. PTV-based IMPT optimization 2020;16:74–80. incorporating planning risk volumes vs robust optimization. Med Phys [42] Kaplan LP, Korreman SS. A systematically compiled set of quantitative metrics 2013;40. to describe spatial characteristics of radiotherapy dose distributions and aid in [22] Wagenaar D, Kierkels RG, Free J, Langendijk JA, Both S, Korevaar EW. treatment planning. Physica Medica 2021;90:164–75. Composite minimax robust optimization of VMAT improves target coverage [43] Miften M, Olch A, Mihailidis D, Moran J, Pawlicki T, Molineu A, et al. Tolerance and reduces non-target dose in head and neck cancer patients. Radiother limits and methodologies for IMRT measurement-based verification QA: Oncol 2019;136:71–7. Recommendations of AAPM Task Group No. 218. Med Phys 2018;45:e53–83. [23] Unkelbach J, Alber M, Bangert M, Bokrantz R, Chan TC, Deasy JO, et al. Robust [44] Hernandez V, Hansen CR, Widesott L, Bäck A, Canters R, Fusella M, et al. What radiotherapy planning. Phys Med Biol 2018;63. 22TR02. is plan quality in radiotherapy? The importance of evaluating dose metrics, [24] Unkelbach J, Chan TCY, Bortfeld T. Accounting for range uncertainties in the complexity, and robustness of treatment plans. Radiother Oncol optimization of intensity modulated proton therapy. Phys Med Biol 2020;153:26–33. 2007;52:2755–73. [45] Hansen CR, Crijns W, Hussein M, Rossi L, Gallego P, Verbakel W, et al. [25] Zhang X, Rong Yi, Morrill S, Fang J, Narayanasamy G, Galhardo E, et al. Robust Radiotherapy Treatment plannINg study Guidelines (RATING): A framework optimization in lung treatment plans accounting for geometric uncertainty. J for setting up and reporting on scientific treatment planning studies. Appl Clin Med Phys 2018;19:19–26. Radiother Oncol 2020;153:67–78. [26] Noufal MP, Widesott L, Sharma SD, Righetto R, Cianchetti M, Schwarz M. The [46] Masi L, Hernandez V, Saez J, Doro R, Livi L. Robotic MLC-based plans: A study of role of plan robustness evaluation in comparing protons and photons plans - plan complexity. Med Phys 2021;48:942–52. an application on IMPT and IMRT plans in skull base chordomas. J Med Phys [47] Landberg T, Chavaudra J, Dobbs J, Gerard J-P, Hanks G, Horiot J-C, et al. ICRU 2020;45:206–14. Report 62. J ICRU 1999;os32:NP–P. [27] Korevaar EW, Habraken SJM, Scandurra D, Kierkels RGJ, Unipan M, Eenink [48] Bohoslavsky R, Witte MG, Janssen TM, van Herk M. Probabilistic objective MGC, et al. Practical robustness evaluation in radiotherapy–A photon and functions for margin-less IMRT planning. Phys Med Biol 2013;58:3563–80. proton-proof alternative to PTV-based plan evaluation. Radiother Oncol [49] Perkó Z, van der Voort SR, van de Water S, Hartman CMH, Hoogeman M, 2019;141:267–74. Lathouwers D. Fast and accurate sensitivity analysis of IMPT treatment plans [28] Yock AD, Mohan R, Flampouri S, Bosch W, Taylor PA, Gladstone D, et al. using Polynomial Chaos Expansion. Phys Med Biol 2016;61:4646–64. Robustness analysis for external beam radiation therapy treatment plans: [50] Tommasino F, Widesott L, Fracchiolla F, Lorentini S, Righetto R, Algranati C, describing uncertainty scenarios and reporting their dosimetric consequences. et al. Clinical implementation in proton therapy of multi-field optimization by Pract Radiat Oncol 2019;9:200–7. a hybrid method combining conventional PTV with robust optimization. Phys [29] McGowan SE, Albertini F, Thomas SJ, Lomax AJ. Defining robustness protocols: Med Biol 2020;65. 045002. a method to include and evaluate robustness in clinical plans. Phys Med Biol [51] Healy BJ, Budanec M, Ourdane B, Peace T, Petrovic B, Sanz DE, et al. An IAEA 2015;60:2671–84. survey of radiotherapy practice including quality assurance extent and depth. Acta Oncol 2020;59:503–10. 261