Open access Original research Tools to assess the measurement properties of quality of life instruments: a meta-­review Sonia Lorente ‍ ‍,1,2 Carme Viladrich,1 Jaume Vives,1,3 Josep-­Maria Losilla1,3 To cite: Lorente S, Viladrich C, ABSTRACT Vives J, et al. Tools to assess Strengths and limitations of this study Objective This meta-­review aims to discuss the the measurement properties methodological, research and practical applications of quality of life instruments: ►► The search strategy has been designed to be com- of tools that assess the measurement properties of a meta-­review. BMJ Open prehensive, following the Peer Review of Electronic instruments evaluating health-­related quality of life 2020;10:e036038. doi:10.1136/ Search Strategies guidelines including specific fil- bmjopen-2019-036038 (HRQoL) that have been reported in systematic reviews. ters for finding studies on psychometric properties Design Meta-­review. ►► Prepublication history and of measurement instruments. Methods Electronic search from January 2008 to May additional material for this ►► A total of 246 systematic reviews were included and, 2020 was carried out on PubMed, CINAHL, PsycINFO, paper are available online. To to our knowledge, this meta-­review provides the SCOPUS, WoS, Consensus-­based Standards for the view these files, please visit broadest overview of the most common tools used selection of health Measurement Instruments (COSMIN) the journal online (http://​dx.​doi.​ to assess measurement properties of health-­related org/​10.​1136/​bmjopen-​2019-​ database, Google Scholar and ProQuest Dissertations and quality of life instruments and their relationship with 036038). Theses. measurement standards, scientific traditions and Results A total of 246 systematic reviews were assessed. the intended use of the measures. Received 27 November 2019 Concerning the quality of the review process, some ►► Some of the included systematic reviews poorly Revised 15 June 2020 methodological shortcomings were found, such as poor Accepted 06 July 2020 reported the review process, outcomes and conclu- compliance with reporting or methodological guidelines. sions, and this fact may have led to the loss of some Regarding the procedures to assess the quality of data. measurement properties, 164 (66.6%) of reviewers applied ►► Inclusion of studies published in English only may one tool at least. Tool format and structure differed across have led to language bias. standards or scientific traditions (ie, psychology, medicine and economics), but most assess both measurement properties and the usability of instruments. As far as the results and conclusions of systematic reviews are a specific study population.1 These system- concerned, only 68 (27.5%) linked the intended use of atic reviews provide both a comprehensive the instrument to specific measurement properties (eg, overview of the measurement properties of evaluative use to responsiveness). health instruments and supportive evidence Conclusions The reporting and methodological for the selection of instruments for a specific quality of reviews have increased over time, but there purpose (eg, research, clinical practice, is still room for improvement regarding adherence to predictive).2 3 In this type of systematic © Author(s) (or their guidelines. The COSMIN would be the most widespread employer(s)) 2020. Re-­use review, different authors have evaluated not and comprehensive tool to assess both the risk of bias permitted under CC BY-­NC. No only the methodological quality of their key of primary studies, and the measurement properties of commercial re-­use. See rights phases—namely the search strategy, the bias and permissions. Published by HRQoL instruments for evaluative purposes. Our analysis of other assessment tools and measurement standards risk assessment of the primary studies and BMJ. 1 can serve as a starting point for future lines of work on the the data synthesis—but also whether the Department of Psychobiology and Methodology of Health COSMIN tool, such as considering a more comprehensive measurement properties of the health status Science. Area of Behavioral evaluation of feasibility, including burden and fairness; instruments have been appraised with stan- Science Methodology, expanding its scope for measurement instruments dardised procedures or tools during the data Universitat Autònoma de with a different use than evaluative; and improving its extraction phase.1 2 4 5 However, depending Barcelona, Cerdanyola del assessment of the risk of bias of primary studies. on the measurement standards on which Vallés, Barcelona, Spain PROSPERO registration number CRD42017065232. 2 Pediatric Area, Consorci Sanitari these tools were developed, the approach de Terrassa, Terrassa, Spain to analyse the measurement properties of 3 Sport Research Institute, instruments may vary.6 This could lead to Universitat Autonoma de INTRODUCTION different conclusions and recommendations, Barcelona, Cerdanyola, Spain The systematic reviews of measurement prop- in spite of the effort undertaken by the inter- Correspondence to erties critically appraise the content and national Society for Quality of Life Research Dr Jaume Vives; measurement properties of all instruments to set consensus-­based minimum standards.7 ​Jaume.​Vives@​uab.​cat that assess a certain construct of interest in Besides, according to Rosenkoetter and Tate,6 Lorente S, et al. BMJ Open 2020;10:e036038. doi:10.1136/bmjopen-2019-036038 1 Open access the assessment tools commonly used by clinicians and disease14 were included. Systematic reviews were required researchers to select the appropriate outcome measures to include the full results report, and detailed informa- for specific purposes show a variety of forms and cover tion about the procedures used to assess the measure- a mix of standards related to reporting, methodological ment properties. quality and statistical outcome quality. The aims of this present meta-­review are to: (1) identify Exclusion criteria systematic reviews assessing the measurement properties Systematic reviews exclusively focused on evaluating of health-­related quality of life (HRQoL) instruments; (2) clinical interventions were excluded. Systematic reviews identify the main tools applied to assess their measure- specifically focused on assessing patient-­ reported ment properties; (3) describe the contents of the applied outcomes measures (PROMs) other than HRQoL for tools (validity, reliability, feasibility, etc); (4) identify the specific diseases, clinical conditions or populations, were measurement standards on which these tools were devel- excluded. Systematic reviews that did not report full infor- oped or conform to, comparing their similarities and mation about the procedures to assess the measurement differences and (5) appraise how authors of these system- properties were also excluded (eg, conference abstracts). atic reviews include the assessment of the measurement Study screening quality in their results and conclusions, that is, to what References identified by the search strategy were entered extent conclusions depend on the results of the eval- to Mendeley reference management software, and dupli- uation of the measurement properties, as well as their cates were removed. Titles and abstracts were screened relationship, if any, with the intended use of the HRQoL independently by two reviewers (SL and JV). When deci- instrument (eg, evaluative). sions were unable to be made from title and abstract alone, the full paper was retrieved. Full-­text inclusion criteria were checked independently by two reviewers (SL METHODS and JV). Discrepancies during the process were resolved The protocol of this review8 was prospectively registered. through discussion (with independent reviews of J-­ML We conducted this meta-­review following the Preferred and CV when necessary). Reporting Items for Systematic Reviews and Meta-­Analyses guidelines (PRISMA).9 10 Data extraction Extracted information of each selected systematic review Search strategy and meta-­analysis included general information such as A systematic search was performed in PubMed, US author, year and quality of review process of systematic National Library of Medicine, by National Center for reviews (eg, protocol registration, reporting guidelines Biotechnology Information (NCBI); CINAHL, Cumula- and use of flow chart). Information concerning the main tive Index to Nursing and Allied Health Literature, by identified tools applied to assess the measurement prop- EBSCOhost; PsycINFO, Psychological Information, by erties of HRQoL instruments included the title, intended APA PsycNET; SCOPUS by Elsevier; WoS, Web of Science use, number of items, response categories, instrument CORE, by Thomson Reuters; Consensus-­based Standards assessment criteria and measurement properties assessed. for the selection of Health Measurement Instruments Information on how authors included the assessment of database, by COSMIN Initiative (http://www.​ cosmin.​ the quality of HRQoL in their results and conclusions was nl/); and Google Scholar (up to 400 links). ProQuest also extracted. Authors of eligible studies were contacted Dissertations and Theses Global was used for searching to provide missing or additional data when necessary. grey literature, and search alerts in all databases were set. The search strategy followed the Peer Review of Elec- Study aim tronic Search Strategies guidelines recommendations,11 12 To examine the methodological, research and practical and consisted of three filters composed of search terms applications of the reported tools in systematic reviews for the following: (1) systematic review methodology; that assess the measurement properties of instruments (2) HRQoL instruments and (3) measurement proper- evaluating quality of life within the context of health and ties. The latter filter was developed by the Vrije Univer- disease, that is, HRQoL. sity Medical Center for finding studies on measurement properties of measurement instruments.13 All filters were adapted for all databases. The searches were completed in RESULTS May 2020. Restrictions by language (English) and publi- Search results cation date (from January 2008) were applied (see online Figure 1 shows the results of the search strategy, reported supplementary file 1 for search strings for all databases). according to the PRISMA flow diagram. A total of 4320 references were identified through database searches. Inclusion criteria After removing duplicates, 3055 titles and abstracts were Systematic reviews specifically aiming to report or to screened. After the assessment of 525 full-­ text docu- assess the measurement properties of instruments evalu- ments for eligibility, a total of 246 systematic reviews were ating the quality of life within the context of health and included in the qualitative analysis. These systematic 2 Lorente S, et al. BMJ Open 2020;10:e036038. doi:10.1136/bmjopen-2019-036038 Open access Figure 1 PRISMA flow chart. Flow diagram for search results (from Moher et al9). COSMIN, Consensus-­based Standards for the selection of health Measurement Instruments; HRQOL, health-­related quality of life; PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-­Analyses. reviews covered a wide range of HRQoL instruments, 238 (96.7 %) reported the search strategy; 116 (47.41%) both generic and disease specific. A total of 24 (9.8 %) reported the detailed syntax for one database at least; of the systematic reviews assessed the quality of one 134 (54.4%) made the article selection by two or more measurement property only, such as the conceptual and independent reviewers; 166 (67.5%) used a flow chart measurement model or the content validity (see online to report search outcomes and 132 (53.7%) stated the supplementary file S2 for characteristics and references funding. These last percentages slightly increased when of studies). reducing the time frame to the last 6 years. Reporting and methodological quality of the studies Table 1 shows the reporting and methodological quality Assessment of measurement properties of HRQoL instruments of systematic reviews. Findings showed that 27 (10.9%) of Assessment procedures of measurement properties the reports registered the protocol prospectively, a figure varied considerably. A total of 164 (66.6%) out of 246 that raised to 20.8% when considering the reports from systematic reviews applied one tool at least, that is, a 2014 onwards; 78 (31.7%) followed reporting guidelines published and well-­accepted list of criteria, to rate the such as PRISMA (50.8% the last 6 years); 42 (17.0% since evidence on measurement properties of instruments; 2008; 23.8% for the last 6 years) assessed the reporting 41 (16.6%) applied their own author’s criteria only; 30 and/or the methodological quality of primary studies (12.2%) followed literature recommendations included using recommended guides, such as Standards for the in very highly circulated books or papers only, and 14 Reporting of Diagnostic Accuracy Studies and Quality (5.7%) used an ad hoc checklist of criteria only. A total Assessment of Diagnostic Accuracy Studies, respectively; of 98 (39.8%) systematic reviews did combine different Lorente S, et al. BMJ Open 2020;10:e036038. doi:10.1136/bmjopen-2019-036038 3 Open access Table 1 Reporting and methodological quality of studies Table 1 Continued 2008–2020 2014–2020 2008–2020 2014–2020 N % N % N % N % Protocol registered prospectively ►► No 80 32.5 18 14.1 ►► Yes, PROSPERO 27 10.9 26 20.5 Funding ►► No registered 219 89.1 100 79.3 ►► Reported 132 53.7 69 54.8 Standards of systematic review reporting and/or quality ►► Not reported or not clearly 114 46.3 57 45.2 assessment stated ►► Yes (AMSTAR, PRISMA, 78 31.7 64 50.8 Total 246 100 126 100 QUOROM…) %, percentage; AMSTAR, assessment of multiple systematic ►► No 168 68.3 62 49.2 reviews; n, frequency; PRISMA, Preferred Reporting Items for Standards to assess reporting and/or quality assessment of Systematic Reviews and Meta-­Analyses; PROSPERO, Prospective primary studies Register of Systematic Reviews; QUADAS, Quality Assessment of Diagnostic Accuracy Studies; QUOROM, quality of reporting of ►► Yes (QUADAS, STARD…) 42 17.0 30 23.8 meta-­analysis; STARD, Standards for the Reporting of Diagnostic ►► No 204 83.0 96 76.2 Accuracy Studies. No of databases searched ►► 1–3 96 39.1 50 39.6 procedures. Most usual combinations were the use of two ►► 4–6 107 43.4 61 48.4 tools or one tool and literature recommendations. ►► 7–9 22 8.9 8 6.3 Tools to assess measurement properties of HRQoL instruments ►► ≥10 18 7.3 6 4.7 The first 12 columns of table 2 present the characteris- ►► Not reported 3 1.2 1 0.8 tics for the identified tools used to assess measurement Other sources properties using the last update we are aware of. Tools ►► Official websites/internet 25 10.1 7 5.5 are reported in order of frequency of use, as pointed ►► Virtual libraries 24 9.7 12 9.4 out in the last row of the table: (1) ‘COSMIN’, COSMIN initiative15 16; (2) ‘Quality Criteria for Measurement ►► Google/google scholar 25 10.1 14 11.0 Properties’, Terwee et al17; (3) ‘Attributes and Criteria ►► Scientific journals/thesis 6 2.4 2 1.6 to assess Health Status and Quality of Life Instruments’, Search strategy Scientific Advisory Committee Medical Outcomes Trust Terms, databases, time period (SACMOT)18 19; (4) ‘Health Status Measures in Economic ►► Yes 238 96.7 123 97.6 Evaluation’, Brazier et al20 21; (5) ‘Guidance for Industry PROMs’, Food and Drug Administration (FDA)22 23; (6) ►► No 8 3.3 3 2.4 ‘Evaluating Patient-­ based Outcomes Measures for use Search syntax in clinical trials’, Fitzpatrick et al24 (also known as Fitz- ►► Detailed syntax reported 115 46.7 79 62.7 patrick’s criteria); (7) ‘International Classification of (Truncations, Booleans…) Functioning’ and ‘International Classification of Func- ►► Syntax not reported or 125 50.8 46 36.5 tioning for Children and Youth’, WHO25; (8) ‘Evaluating not detailed enough to be Measures of Patient-­ Reported Outcomes (EMPRO)’, replicable Spanish Cooperative Investigation Network for Health ►► Supplementary file under 5 2.1 1 0.8 and Health Service Outcomes Research26; (9) ‘Spinal request (not available) Cord Injury Criteria’, Spinal Cord Injury Rehabilitation Inclusion/exclusion selection criteria Evidence27 28; (10) ‘Criteria for Assessing the Tools of Disability Outcomes Research’, Andresen29 (also known ►► Reported and well-­defined 229 93.1 122 96.8 as Andresen’s tool); (11) ‘CanChild Outcomes Measures’, ►► Not reported or not clearly 17 6.9 4 3.2 CanChild Center for Childhood Disability Research30 stated and (12) ‘Outcomes Measures in Rheumatology Clin- Article selection ical Trials (OMERACT)’, OMERACT initiative.31 Table 2 ►► By two or more independent 134 54.4 87 69.0 also includes a final column showing the characteristics reviewers of Testing Standards by American Educational Research ►► Not reported or not clearly 112 45.6 39 31.0 Association, American Psychological Association and stated National Council on Measurement in Education32 33 Flow chart (hereinafter ‘Testing Standards’) initially published in ►► Yes 166 67.5 108 85.7 1954 and regularly updated every decade using consensus based procedures. The Testing Standards are the source Continued of most of the technical vocabulary for measurement 4 Lorente S, et al. BMJ Open 2020;10:e036038. doi:10.1136/bmjopen-2019-036038 Table 2 Tools to assess measurement properties. characteristics and comparison to testing standards Terwee’s Attributes and Economic Guidance for Fitzpatrick’s Canchild Tools Cosmin criteria criteria evaluation industry criteria ICF ICFCY EMPRO SCI criteria Andresen’s tool outcomes Omeract Testing standards Development Delphi Author criteria Expert panel Literature Consensus Literature   Expert panel Expert panel Literature Expert panel Expert panel Consensus literature Delphi Sponsor/s COSMIN Author SACMOT Standing FDA staff Standing WHO IRYSS SCIRE working Author CanChild centre OMERACT AERA, APA, NCME initiative working group group of group of health member committee group staff initiative health technology states technology Approval 2010, 2018 2007 1996, 2002, 2013 1999, 2017 2006, 2009 1998 2001, 2008 2008, 2016 2000 1987†, 2004 1992, 1954, 1966, 1974, updates 2019* 1998,2007,2014, 1985, 1999, 2014 2019 Items (scoring) 5–18 items/box 8–9 items total Not item Not item Not item Not item Not item 39 items(strongly 3–5 items/box Eleven items 2–6 items/ 2–5 items/box Not item structured (+/−/?) (+/−/?) structured (no structured structured (no structured (no structured agree, agree, (++++/+++/++/+) total (A, B, C) box (excellent, (Green, amber, (no scoring) scoring) (no scoring) scoring) scoring) (no disagree, adequate, poor) red, white) scoring) strongly disagree) Measurement Content Content Conceptual and Descriptive Conceptual Use Content/ Content Conceptual and Content criterion Conceptual and Use scale Content, face Content response properties construct construct measurement (Content model face construct measurement (concurrent measurement construction construct process Int. structure (Int. structure (Hypotheses model Face Content (convergent, model predictive model Content (Convergent, (Dimensions, DIF) Validity cross-­cultural test) Content Construct) construct discriminant, int. Content ‘discriminant’) Instrument bias construct divergent) Relations to other hypotheses test) Criterion (Gold construct Preference-­ (Hypothesis test, structure) construct Clinical utility Int. structure (Hypotheses Criterion variables (Hypotheses Criterion (Gold standard) (Hypotheses test) based discriminant, Criterion (Hypotheses (consequential convergent test) (Accuracy) test,Convergent, standard) Floor/Ceiling Criterion (Gold valuation convergent, (Predictive) test) validity) discriminant Criterion (Gold Discrimination Discriminant, criterion, standard) Empirical known groups) Cut-­score Criterion Floor/Ceiling standard) (Sensitivity over responsiveness Lorente S, et al. BMJ Open 2020;10:e036038. doi:10.1136/bmjopen-2019-036038 (Criterion) precision time and over Consequences Responsiveness   Responsiveness treatment) Responsiveness Responsiveness Responsiveness Responsiveness Responsiveness Responsiveness Responsiveness   Reliability Int. consistency Int. consistency Int. consistency Test retest Test retest Int. consistency   Int. consistency Int. consistency Int. consistency Int. consistency Reproducibility Int. consistency test measurement reproducibility reproducibility Inter-­rater Inter-­rater Int. reproducibility reproducibility test retest test retest intra/inter-­rater test retest retest alternate forms error (Test retest, (Agreement, (Test retest, inter-­ consistency (Test retest) (Test retest, test retest scorers and decision agreement) relative rater) inter-­rater) consistency/accuracy measurement error)   Fairness                         Equivalence of accommodations Other                 Norms Norms, standard Norms   Scales, norms, Score characteristics values standardisation comparability   Interpretability Interpretability Interpretability   Interpretability Interpretability   Interpretability         Test development and revision       Burden   Burden Acceptability   Burden Burden Burden       (Burden)       Administration   Administration     Administration Administration Administration       accessible forms accessible forms accessible forms accessible forms   Feasibility   Cultural Practicality   Feasibility cultural   Cultural Applicability Cultural Clinical utility Feasibility   adaptations adaptations adaptations cultural adaptations (Feasibility) adaptations Frequency of 61 (30.4) 45 (22.4) 33 (16.4) 17 (8.4) 14 (6.9) 14 (6.9) 7 (3.4) 4 (2.0) 2 (1.0) 2 (1.0) 1 (0.5) 1 (0.5) 0 use (%) *Updated version at website. †Reference at 2004. AERA, american educational research association; APA, American Psychological Association; COSMIN, Consensus-­based Standards for the selection of health Measurement Instruments; DIF, differential item functioning; EMPRO, Evaluating Measures of Patient Reported Outcomes; FDA, Food and Drug Administration; ICF, international classification of functioning; ICFCY, international classification of functioning for children and youth; IRYSS, Investigation Network for Health and Health Service Outcomes Research; NCME, National Council on Measurement in Education; OMERACT, Outcomes Measures in Rheumatology Clinical Trials; SACMOT, Scientific Advisory Committee Medical Outcomes Trust; SCI, spinal cord injury; SCIRE, Spinal Cord Injury Rehabilitation Evidence.; Open access 5 Open access properties in HRQoL instruments, therefore, they will be in Testing Standards where it is included as a particular used as a reference to compare the twelve identified tools. case of differential item functioning when assessing the In fact, these standards have already been recommended internal structure of the instruments (see online supple- to establish a unified approach to validity and reliability mentary file S3 for more details). of results derived from psychometric instruments in clin- ical medicine, research and education.34 Intended uses of instruments and their association to Different methodologies were used to develop the tools. measurement properties The expert panel consensus and the literature review Some of the differences between tools can be attributed were the most usual methods, led by steering committees to the fact that they are devoted to the evaluation of or staff/working groups. The format and structure of instruments developed with different intended uses. For these tools also vary. Whereas seven of them were itemised instance, COSMIN aims at assessing the quality of instru- to allow the assignment of quality scores, the other six ments for an evaluative purpose whereas the Economic took the form of standards or guidelines. Tools with an Evaluation tool aims at the assessment of instruments for itemised structure were the COSMIN, Quality Criteria for analytical purposes. Nevertheless, the relation between Measurement Properties, EMPRO, SCI Criteria, Criteria the intended use of the instruments and the measurement for Assessing the Tools of Disability Outcomes Research properties assessed is not usually included in the conclu- (Andresen’s tool), CanChild Outcomes Measures and sions of the systematic reviews. Table 3 shows the intended OMERACT. use of instruments, based on the framework proposed Among all measurement properties considered in by McDowell et al35 and the association to measurement Testing Standards, 11 out of the 12 tools recommended properties that reviewers established in their conclusions. to assess the conceptual and measurement model; The instruments were most frequently used for evalua- content, structural, convergent, discriminant, concur- tion (178, 72.3%) and for assessment of impact of disease rent and predictive validity; responsiveness or sensitivity on HRQoL (138, 55.1%), either alone or in conjunction. to change; and internal consistency, test–retest and inter-­ Other purposes were analytic (35, 14.2%), diagnostic (16, rater reliability. However, the approach to analyse these 6.5%), descriptive (4, 1.6%) and predictive (2, 0.8%). measurement properties varied, with examples found A total of 6 (2.4%) systematic reviews did not report or in construct validity, criterion validity and reliability. did not clearly state the intended use of the instruments. Depending on the tool, the validity of the construct As far as the assessment and conclusions is concerned, can be evaluated either by hypothesis confirmation in only 68 (27.6%) systematic reviews linked the intended general (eg, COSMIN or EMPRO), or by specific hypoth- use of the instrument to measurement properties. The esis based on correlations with other measures, that is, most common use was evaluative, generally associated to convergent and discriminant validity (eg, Andresen’s tool). Criterion validity can be assessed either exclu- responsiveness, content validity or reliability, for example. sively by calculating the correlation coefficient with a When the purpose was the assessment of the impact of gold standard (eg, CanChild Outcomes Measures) or by disease on HRQoL, the conceptual and measurement obtaining variously correlation, specificity and sensitivity model and content validity were usually reported. The or predictive values (eg, FDA). Reliability can be anal- analytical purpose involved reporting preference-­based ysed either by test retest reliability, inter-­rater reliability valuation (eg, utility scores) and evidence of agreement, and internal consistency (eg, FDA), or only by test retest and the diagnostic use was linked to known groups and inter-­rater agreement (eg, Economic evaluation). validity and test–retest reliability. To better understand Despite the Testing Standards recommendations, just these results, some examples are given. First, the evalu- one tool includes additional criteria to assess consequen- ative purpose was associated to responsiveness, we found tial validity (SCI), and four assess fairness (eg, accessible conclusions such as: ‘For use in longitudinal studies or forms for subjects with vision impairment or for specific clinical practice, where responsiveness is an issue, the populations) (SACMOT, FDA, SCI and Andresen’s tool). Minnesota Living with Heart Failure Questionnaire None of them includes criteria to assess the validity of and the Chronic Heart Failure Questionnaire would be response processes. Other HRQoL instrument character- adequate’.36 Second, the intended use was the assessment istics, such as feasibility (eg, cost of obtaining a sample), of the impact of disease on HRQoL, the usual association acceptability (eg, suitability from the patient perspective) was to the measurement model and conclusions resem- or burden (eg, the time or effort placed on the admin- bled this one: ‘None of the RLS specific QOL measures istration of the instrument) are assessed instead. Finally, appears to have been informed by a conceptual model notice that some concepts have changed their place or a conceptual framework. Consequently, none can be over time. The clearest case is evidence regarding cross-­ considered comprehensive in terms of assessing the full cultural equivalence, which was treated as an additional impact of Rest Legs Syndrome on QOL’.37 Third, an characteristic of the instruments in most tools released example illustrating general conclusions, that is, conclu- before 2014 (eg, EMPRO or SCI), but was considered a sions that did not associate the intended use of the proper measurement property in the COSMIN’s 2018 instrument to any specific measurement properties, was update. It is also considered a measurement property as follows: ‘None of the available instruments fulfils the 6 Lorente S, et al. BMJ Open 2020;10:e036038. doi:10.1136/bmjopen-2019-036038 Open access process, describe the most used tools to assess measure- Table 3 Intended use of instruments and their association to measurement properties ment properties and examine how reviewers included the assessment of the quality of HRQoL in their conclusions. Intended use of instruments identified across the systematic reviews Frequency % (over 246) Reporting and methodological quality of the studies Evaluative (Change scores pre and 178 72.3 Findings showed how the reporting and methodological poststudies. Effectiveness of an quality of systematic reviews has increased over time. Most intervention) reviewers reported the search strategy, stated the inclu- Impact of disease on HRQoL (disease 138 55.1 sion and exclusion criteria taking the judgement of two symptoms, burden…) or more independent reviewers into account and used Analytic (health policies. Cost-­ 35 14.2 a flow chart to report search outcomes. However, some effectiveness. Funding) crucial methodological shortcomings were found. Prac- Diagnostic (Distinguish between 16 6.5 tices such as registration of the protocol, reporting the groups, levels of severity…) detailed search syntax for one database at least, adherence Descriptive (Health measures in 4 1.6 to reporting guidelines, and assessing the reporting and surveys. Needs of groups of people) the methodological quality of primary studies were quite Predictive (Anticipation of future health 2 0.8 sparse even in recent years. As Pussegoda et al4 suggested, status. Risk factors. Risk profiles) this fact may be related to the percieved time-­consuming Intended use is no reported or no 6 2.4 task of using guidelines or to the lack of information clearly stated about the most appropiate tool. According to our data, Conclusions according to the n % (over 246) there is still large room for improvement in the assess- intended use of instruments ment of the methodological quality of included studies Yes, reviewers made specific 68 27.6 in order to attend to Terwee et al’s warning2 of avoiding conclusions the risk of presenting biased results, leading to underesti- No, reviewers made general 178 72.4 conclusions mation or overestimation of the quality of an instrument. Measurement properties associated n % (Over 68) to the intended use of the Assessment of measurement properties of HRQoL instruments instrument Assessment procedures of measurement properties of Evaluative HRQoL instruments were diverse. Most of the reviewers used at least one tool. Nevertheless, there were reviewers Responsiveness/Conceptual and 41 60.3 Measurement Model/Content validity/ that applied their own criteria, followed literature recom- Reliability (internal consistency, mendations or applied different ad hoc devised check- test retest)/Respondent Burden/ lists. The use of such diverse procedures is noticeable, Convergent validity/Cross cultural even in recent years, when well-­accepted tools to assess validity measurement properties are available. Impact Our meta-­review identified up to twelve tools. Seven of  Conceptual and Measurement 29 42.6 them had an itemised structure, offering a comparable Model/Content validity approach to rate the evidence on measurement proper- Analytic ties. Length and scoring differed, but also the instrument  Preference-­based valuation/ 11 16.2 assessment criteria. Actually, depending on the tool used, agreement the approach to assess properties varied greatly, with Diagnostic potentially serious consequences. The fact that a single  Known groups validity/test–retest 7 10.3 measurement property is or isn’t required can change Predictive the status of quality of the evidence supporting the same  Sensivity and specificity 1 1.5 measurement instrument. The variety of forms found were in concordance to results from related research, which (%), percentage. HRQoL, health-­related quality of life. also highlighted the complexity with regard to defini- tions of measurement properties.6 This complexity is also reflected in the search filter developed by the COSMIN initiative.13 They recommend using three filters that sum psychometric demands of reliability, validity and respon- up more than 100 search terms in order to get sensible siveness to serve as a primary outcome measure in clinical and specific results. In addition, and also depending on trials’.38 the tool used, other characteristics, such as feasibility, acceptability and burden were assessed. In spite of the DISCUSSION diversity, a shared conclusion can be stated as follows: The present meta-­ review identified 246 systematic because these instruments are to be used in the daily reviews assessing measurement properties of HRQoL practice, their usability should be always balanced with instruments in order to analyse the quality of the review other characteristics considered as proper measurement Lorente S, et al. BMJ Open 2020;10:e036038. doi:10.1136/bmjopen-2019-036038 7 Open access properties.39 40 For instance, an instrument needs to be those used in calculating quality-­adjusted life years and long enough to ensure reliability and construct validity, using preference-­ based methods to obtain their data. but short enough to ensure the adequate response rate Due to that, some very popular measurement proper- and sample size. Otherwise the instrument intended use ties such as internal structure based on factor analysis and sustainability will be at hazard.39 are not relevant and thus not considered in their tools. The differences between tools and their potentially In this tradition, the main concern regarding the integ- serious consequences on the assessment of the quality of rity of the instrument purpose is whose values should the primary studies may be better addressed in the light be considered when determining preferences and how of three considerations: the date of publication, the main well the preferences of patients and decision makers are scientific tradition involved when developing the tools, likely to conform to the main assumptions of the utility and the intended uses of the instruments under assess- models.20 21 ment. Some differences can be simply explained by the date of publication of the tools. As an example, where Intended uses of instruments and their association to older tools require specific forms of validity evidence measurement properties related to external variables such as convergent and In our view, considering in the first place the intended discriminant validity, recent tools incorporate the more use of the HRQoL instrument would help to reconcile general view of hypothesis testing. That is, when devel- the different requirements included in each tool. Tools oping a new use for an instrument, hypotheses should be for evaluating the measurement quality of instruments made regarding the expected relations with other rele- should be adapted or extended according to the different vant variables in their nomological network and these intended uses of these instruments, such as evaluative, hypotheses and no other should be tested.32 Regarding impact of disease, analytic, diagnostic, descriptive or the scientific traditions, the assessment of outcomes is predictive. Notice that depending on the intended use a constitutive part of the disciplines of Education and of the measure, some domains of validity and reliability Psychology where the Testing Standards come from. In may be of greater or lesser relevance.6 16 For instance, these contexts, participation is taken for granted as assess- an instrument developed to assess longitudinal changes ment practices result in high stakes decisions such as, for should demonstrate high responsiveness,6 but if used instance, certification or personnel selection. The main for diagnostic purposes, it should be able to distinguish concern regarding integrity of the instrument purpose among individuals or groups,6 that is, known groups is its fakeability, which could distort the decision-­making validity. Another example is the internal consistency reli- process, and this would explain the interest in response ability based on interitem relationships that may be not processes in this field.41 42 By contrast, the main objec- relevant for a preference-­based instrument but is relevant tive in the discipline of Medicine is to provide health- for an instrument based on a unidimensional measure- care services. Evaluation of subjective views of patients ment model. However, our data showed that only a few was a late addition related to the inclusion of HRQoL authors established a clear link in their recommenda- in the accounting of healthcare outcomes, despite the tions between the intended use of the measure and the instruments assessing the patient experience should be reported evidence of measurement properties. The vast acceptable to both patients and clinicians, as Beattie et field of HRQoL offered a plethora of instruments but, al highlighted.39 Specifically, in the context of disability as most reviewers did not take the intended use of the research, the administrative and respondent burden instrument into account, the overall rating of measure- requires additional consideration. The administrative ment properties was not consistent and thus the instru- burden may include the need for a Sign Language inter- ment may or may not have been adequate for its intended preter, and the respondent burden includes the length of use. Because the evaluation and improvement of quality the questionnaire, which is especially relevant when using of life is considered a public health priority,14 we strongly HRQoL instruments with cognitively impaired subjects.29 encourage researchers to assess the quality of measure- Balancing the traditional psychometric criteria, the prac- ment properties of HRQoL instruments according to the ticalities of the instruments and patient preferences intended use of the measure. Otherwise, there is a serious is a generic recommendation for health research, but risk of biased results, which could lead to underrating the becomes a special obligation for research with people with quality and suitability of the instrument. specific needs.29 Moreover, devising test accommodations or accessible forms when needed is expected to become a required psychometric criterion in the near future, given CONCLUSIONS that it has already been included under the title ‘fairness The quality of the systematic review process has been in testing’ as a new section next to validity and reliability increasing over time, but it should still improve with in the chapter of measurement foundations in the most regard to the prospective registration of protocol, and recent update of Testing Standards.32 with respect to the adoption of guidelines to improve Another criterion is that of economic evaluation, tradi- both the methodological and reporting quality of the tionally embedded in providing quantitative judgements reviews. In the specific context of systematic reviews of able to be integrated into mathematical models such as measurement instruments, enhancing the quality of the 8 Lorente S, et al. BMJ Open 2020;10:e036038. doi:10.1136/bmjopen-2019-036038 Open access process also involves the assessment of measurement us to suggest future lines of work on this tool. First, the properties by using a standardised tool. The selection of current format of COSMIN is fairly complex, requiring the most suitable tool may be addressed according to the high expertise in the field of psychometrics and specific coverage of the appraised measurement properties, but training for its proper application. The reporting of the also according to other important criteria, such as the inter-­rater agreement coefficients when reviewers use the intended use of the HRQoL instruments, the format of last version of COSMIN may provide useful data about the tool and whether it assesses both usability (eg, feasi- its reliability. Second, consideration should be given to bility or burden) and accommodation (or accessible the testing standards recommendation on the inclusion forms). First, the assessment methodology should be of the assessment of fairness (ie, evaluation of accessible adapted when necessary, establishing the relation between forms for specific populations). Third, the feasibility the intended use of the HRQoL instruments and the of the measurement instruments, merely described in measurement properties assessed. Second, to standardise COSMIN, and their burden, should be properly rated, the review process, the tool’s format should be itemised with examples found in EMPRO or Andresen’s tool. offering a comparable approach to rate the evidence on Fourth, it must be considered that the RoB evaluation of measurement properties. Those tools that take the form studies is itself a productive field of research with a long of guidelines, such as the SACMOT or the economic eval- tradition, with specific tools that have been developed for uation would be considerably upgraded if the structure is reconverted, since the current format only allows descrip- different research questions and study designs. Examples tion rather than critical appraisal of the quality of an might be found in the Cochrane Collaboration’s Tool for instrument, and furthermore, it complicates comparison Assessing the Risk of Bias of Clinical Trials,46 the Newcastle of results. Lastly, because systematic reviews on measure- Ottawa Scale47 for non-­randomised studies, or the Quality ment properties aim to help professionals to select the best Assessment Tool for Cohort Studies.48 49 From our point instrument for a clinical scenario, the feasibility, patient’s of view, the COSMIN proposal could also be simplified preferences, administrator and respondent burden, and and improved by guiding the reviewers towards the iden- the accommodations (or accessible forms) should be tification of the most appropriate RoB assessment tools addressed and evaluated. Otherwise the suitability and instead of developing their own RoB appraisal guidelines, the intended use of instruments might be compromised, taking advantage of knowledge and innovations in that especially in the context of disability research. Tools iden- field of research. tified in our meta-­review that meet most of these criteria And last, but not least, improving the quality of system- are the COSMIN, EMPRO, SCI criteria, Andresen’s tool, atic reviews encompasses researchers, sponsors and CanChild Outcomes and OMERACT, since all of them promoters, but also journals, which should require full cover a wide range of measurement properties, offer an compliance with reporting and methodological guide- item structure, and assess the usability of instruments. lines, and the use of assessment tools. Special mention is due to the COSMIN, the most wide- spread and comprehensive tool to assess measurement Twitter Jaume Vives @VivesJ_Research and Josep-­Maria Losilla @jmlosilla properties of health instruments designed for an evalu- Contributors All authors meet the criteria recommended by the International ative purpose. The COSMIN standards were developed Committee of Medical Journal Editors, ICMJE. All authors made substantial in a Delphy study43 aiming to improve the selection contributions to conception and design, piloted the inclusion criteria and provided the direction of the data extraction and analysis. SL drafted the article and JV, CV of the most appropriate health instrument for a clin- and J-­ML critically revised the draft for important intellectual content. All authors ical scenario. The most recent version of the COSMIN agreed on the final version. consists of a manual for conducting systematic reviews Funding This work was supported by the Grant PGC2018-100675-­B-­I00, Spanish of health instruments, providing different steps with Ministry of Science, Innovation and Universities (Spain). respect to the literature search process, the assessment Competing interests None declared. of measurement properties and feasibility of instruments, Patient and public involvement Patients and/or the public were not involved in and the evaluation of the risk of bias (RoB) of studies the design, or conduct, or reporting, or dissemination plans of this research. according to the Cochrane methodology.16 Additionally, Patient consent for publication Not required. the COSMIN initiative recently developed a guideline Provenance and peer review Not commissioned; externally peer reviewed. exclusively focused on assessing the content validity of Data availability statement Data are available on reasonable request. Data are health instruments, considered the most important prop- available on reasonable request to the authors. erty to ensure the adequate reflection of the construct Open access This is an open access article distributed in accordance with measured.44 45 In the light of these considerations, we the Creative Commons Attribution Non Commercial (CC BY-­NC 4.0) license, strongly recommend the application of the latest version which permits others to distribute, remix, adapt, build upon this work non-­ of the COSMIN to conduct high-­quality systematic reviews commercially, and license their derivative works on different terms, provided the on measurement properties of health instruments for an original work is properly cited, appropriate credit is given, any changes made evaluative purpose, or for other purposes with appro- indicated, and the use is non-­commercial. See: http://​creativecommons.​org/​ licenses/​by-​nc/​4.​0/. priate adaptation. Despite COSMIN’s many strengths, our analysis of the ORCID iD other assessment tools and measurement standards allow Sonia Lorente http://​orcid.​org/​0000-​0002-​5494-​3325 Lorente S, et al. BMJ Open 2020;10:e036038. doi:10.1136/bmjopen-2019-036038 9 Open access REFERENCES 24 Fitzpatrick R, Davey C, Buxton MJ, et al. Evaluating patient-­based 1 Mokkink LB, Terwee CB, Stratford PW, et al. Evaluation of the outcome measures for use in clinical trials. Vol. 2, 1998. methodological quality of systematic reviews of health status 25 World Health Organization. International classification of functioning measurement instruments. Qual Life Res 2009;18:313–33. (ICF), 2016. Available: www.​who.​int/​classifications/​icf/​en/ [Accessed 2 Terwee CB, Prinsen CAC, Ricci Garotti MG, et al. The quality 6 Feb 2020]. of systematic reviews of health-­related outcome measurement 26 Valderas JM, Ferrer M, Mendívil J, et al. Development of EMPRO: a instruments. Qual Life Res 2016;25:767–79. tool for the standardized assessment of patient-­reported outcome 3 Prinsen CAC, Mokkink LB, Bouter LM, et al. COSMIN guideline for measures. Value Health 2008;11:700–8. systematic reviews of patient-­reported outcome measures. Qual Life 27 SCIRE. Spinal cord injury rehabilitation evidence. Available: https://​ Res 2018;27:1147–57. scireproject.​com 4 Pussegoda K, Turner L, Garritty C, et al. Identifying approaches 28 Johnston MV, Graves DE. Towards guidelines for evaluation of for assessing methodological and reporting quality of systematic measures: an introduction with application to spinal cord injury. J reviews: a descriptive study. Syst Rev 2017;6:1–12. Spinal Cord Med 2008;31:13–26. 5 Pussegoda K, Turner L, Garritty C, et al. Systematic review 29 Andresen EM. Criteria for assessing the tools of disability outcomes adherence to methodological or reporting quality. Syst Rev research. Arch Phys Med Rehabil 2000;81:S15–20. 2017;6:1–14. 30 Law M. Outcome measures rating form guidelines, 2004. Available: 6 Rosenkoetter U, Tate RL. Assessing features of psychometric https://www.​canchild.​ca/​system/​tenon/​assets/​attachments/​000/​000/​ assessment instruments: a comparison of the COSMIN checklist 371/​original/​measguid.​pdf with other critical appraisal tools. Brain Impair 2017:1–16. 31 OMERACT. Instrument selection for core outcome measurement 7 Reeve BB, Wyrwich KW, Wu AW, et al. ISOQOL recommends sets. OMERACT Handbook, 2019. https://​omeracthandbook.​org/​ minimum standards for patient-­reported outcome measures used in handbook patient-­centered outcomes and comparative effectiveness research. 32 APA, AERA, NCME. Standards for educational and psychological Qual Life Res 2013;22:1889–905. testing. American Educational Research Association, 2014. 8 Lorente S, Vives J, Viladrich C, et al. Tools to assess the 33 APA, AERA, NCME. Standards for educational and psychological measurement properties of quality of life instruments : a meta-­review testing. American Educational Research Association, 1999. protocol. BMJ Open 2018;8:1–4. 34 Cook DA, Beckman TJ. Current concepts in validity and reliability 9 Moher D, Liberati A, Tetzlaff J, et al. Preferred reporting items for for psychometric instruments: theory and application. Am J Med systematic reviews and meta-­analyses: the PRISMA statement. Ann 2006;119:166.e7–166.e16. Intern Med 2009;151:264–9. 35 McDowell I, Spasoff RA, Kristjansson B. On the classification 10 Liberati A, Altman DG, Tetzlaff J, et al. The PRISMA statement for of population health measurements. Am J Public Health reporting systematic reviews and meta-­analyses of studies that 2004;94:388–93. evaluate health care interventions: explanation and elaboration. Ann 36 Garin O, Ferrer M, Pont A, et al. Disease-­specific health-­related Intern Med 2009;151:W65–94. quality of life questionnaires for heart failure: a systematic review with 11 McGowan J, Sampson M, Salzwedel DM, et al. PRESS peer review meta-­analyses. Qual Life Res 2009;18:71–85. of electronic search strategies: 2015 guideline explanation and 37 Speight J, Howarth A. Quality of life in restless legs syndrome: a elaboration (PRESS E&E). Cadth Methods Guidel 2016:40–6. systematic review of clinical trials and a critical review of instruments. 12 McGowan J, Sampson M, Salzwedel DM, et al. PRESS peer review Patient 2010;3:185–203. of electronic search strategies: 2015 guideline statement. J Clin 38 Chassany O, Holtmann G, Malagelada J, et al. Systematic Epidemiol 2016;75:40–6. review: health-­related quality of life (HRQOL) questionnaires in 13 Terwee CB, Jansma EP, Riphagen II, et al. Development of a gastro-­oesophageal reflux disease. Aliment Pharmacol Ther methodological PubMed search filter for finding studies on 2008;27:1053–70. measurement properties of measurement instruments. Qual Life Res 39 Beattie M, Lauder W, Atherton I, et al. Instruments to measure patient 2009;18:1115–23. experience of health care quality in hospitals: a systematic review 14 Secretary’s Advisory Committee on National Health Promotion and protocol. Syst Rev 2014;3:4. Disease Prevention Objectives. Health-­related quality of life and well-­ 40 Lorente S, Losilla J-­M, Vives J. Instruments to assess patient being, 2010. Available: https://www.​healthypeople.​gov/​sites/​default/​ comfort during hospitalization: a psychometric review. J Adv Nurs files/​HRQoLWBFullReport.​pdf 2018;74:1001–15. 15 Mokkink L, Terwee C, Patrick D, et al. The COSMIN checklist manual, 41 Ferrando PJ, Anguiano-­Carrasco C. A structural model-­based 2012. Available: http://www.​cosmin.​nl/​images/​upload/​files/​COSM​ optimal Person-­Fit procedure for identifying Faking. Educ Psychol INch​eckl​istm​anualv9.​pdf Meas 2013;73:173–90. 16 Mokkink LB, Prinsen C, Patrick D, et al. COSMIN methodology for 42 Ferrando PJ, Anguiano-­Carrasco C. A structural equation model at systematic reviews of patient-­reported outcome measures (PROMs). the individual and group level for assessing faking-­related change. User manual 2018:1–78. Struct Equ Model 2011;18:91–109. 17 Terwee CB, Bot SDM, de Boer MR, et al. Quality criteria 43 Mokkink LB, Terwee CB, Patrick DL, et al. The COSMIN checklist were proposed for measurement properties of health status for assessing the methodological quality of studies on measurement questionnaires. J Clin Epidemiol 2007;60:34–42. properties of health status measurement instruments: an 18 Aaronson N, Alonso J, Burnam A, et al. Assessing health status and international Delphi study. Qual Life Res 2010;19:539–49. quality-­of-­life instruments: attributes and review criteria. Qual Life 44 Terwee CB, Prinsen CA, Chiarotto A. COSMIN methodology for Res 2002;11:193–215. assessing the content validity of PROMs: user manual. Vol. 120, 19 Lohr KN, Aaronson NK, Alonso J, et al. Evaluating quality-­of-­life and 2018. health status instruments: development of scientific review criteria. 45 Terwee CB, Prinsen CAC, Chiarotto A, et al. COSMIN methodology Clin Ther 1996;18:979–92. for evaluating the content validity of patient-­reported outcome 20 Brazier J, Deverill M, Green C. A review of the use of health measures: a Delphi study. Qual Life Res 2018;27:1159–70. status measures in economic evaluation. Health Technol Assess 46 Higgins JPT, Altman DG, Gøtzsche PC, et al. The Cochrane 1999;3:1–164. collaboration's tool for assessing risk of bias in randomised trials. 21 Brazier J, Ratcliffe J. Measurement and Valuation of Health for BMJ 2011;343:d5928–9. Economic Evaluation. In: International encyclopedia of public health. 47 Wells G, Shea B, O'Connell D, et al. The Newcastle-­Otawa scale . Elsevier, 2017: Vol. 4. 586–93. (NOS) for assessing the quality of nonrandomised studies in 22 Department of Health and Human Services. Guidance for industry meta-­analyses, 2000. Available: www.​ohri.​ca/​programs/​clinical_​ patient-­reported outcome measures: use in medical product epidemiology/​oxford.​asp development to support labeling claims: draft guidance. Vol. 20, 48 Jarde A, Losilla J-­M, Vives J, et al. Q-­Coh: A tool to screen the 2006. methodological quality of cohort studies in systematic reviews and 23 Department of Health and Human Services. Guidance for industry meta-­analyses. Int J Clin Heal Psychol 2013;13:138–46. patient-­reported outcome measures: use in medical product 49 Jarde A, Losilla J-­M, Oliveras I. Quality assssment tool for cohort development to support labeling claims, 2009. studies (Q-­COH II) user’s manual, 2014: 1–13. 10 Lorente S, et al. BMJ Open 2020;10:e036038. doi:10.1136/bmjopen-2019-036038