ORIGINAL RESEARCH published: 07 August 2020 doi: 10.3389/fneur.2020.00632 Harnessing Real-World Data to Inform Decision-Making: Multiple Sclerosis Partners Advancing Technology and Health Solutions (MS PATHS) Ellen M. Mowry 1† , Robert A. Bermel 2† , James R. Williams 3 , Tammie L. S. Benzinger 4 , Carl de Moor 3 , Elizabeth Fisher 3 , Carrie M. Hersh 5 , Megan H. Hyland 6 , Izlem Izbudak 1 , Stephen E. Jones 2 , Bernd C. Kieseier 3 , Hagen H. Kitzler 7 , Lauren Krupp 8 , Yvonne W. Lui 8 , Xavier Montalban 9 , Robert T. Naismith 4 , Jacqueline A. Nicholas 10 , Fabio Pellegrini 3 , Edited by: Alex Rovira 9 , Maximilian Schulze 11 , Björn Tackenberg 11 , Mar Tintore 9 , Marcello Moccia, Madalina E. Tivarus 6 , Tjalf Ziemssen 7 and Richard A. Rudick 3* University of Naples Federico II, Italy 1 Reviewed by: Johns Hopkins University, Baltimore, MD, United States, 2 Cleveland Clinic, Cleveland, OH, United States, 3 Biogen, Pietro Iaffaldano, Cambridge, MA, United States, 4 Washington University in St. Louis, St. Louis, MO, United States, 5 Cleveland Clinic Lou University of Bari Aldo Moro, Italy Ruvo Center for Brain Health, Las Vegas, NV, United States, 6 University of Rochester Medical Center, Rochester, NY, Kimberley Allen-Philbey, United States, 7 Center of Clinical Neuroscience, University Clinic Carl Gustav Carus, TU Dresden, Dresden, Germany, 8 Barts Health NHS Trust, New York University, New York, NY, United States, 9 Vall d’Hebron University Hospital, Barcelona, Spain, 10 OhioHealth, United Kingdom Columbus, OH, United States, 11 University Hospital of Giessen and Marburg, Marburg, Germany *Correspondence: Richard A. Rudick Background: Multiple Sclerosis Partners Advancing Technology and Health Solutions richard.rudick@biogen.com (MS PATHS) is the first example of a learning health system in multiple sclerosis (MS). This † These authors share first authorship paper describes the initial implementation of MS PATHS and initial patient characteristics. Specialty section: Methods: MS PATHS is an ongoing initiative conducted in 10 healthcare institutions This article was submitted to in three countries, each contributing standardized information acquired during routine Neurorehabilitation, a section of the journal care. Institutional participation required the following: active MS patient census of Frontiers in Neurology ≥500, at least one Siemens 3T magnetic resonance imaging scanner, and willingness Received: 20 December 2019 to standardize patient assessments, share standardized data for research, and offer Accepted: 28 May 2020 universal enrolment to capture a representative sample. The eligible participants have Published: 07 August 2020 diagnosis of MS, including clinically isolated syndrome, and consent for sharing Citation: Mowry EM, Bermel RA, Williams JR, pseudonymized data for research. MS PATHS incorporates a self-administered patient Benzinger TLS, de Moor C, Fisher E, assessment tool, the Multiple Sclerosis Performance Test, to collect a structured history, Hersh CM, Hyland MH, Izbudak I, patient-reported outcomes, and quantitative testing of cognition, vision, dexterity, and Jones SE, Kieseier BC, Kitzler HH, Krupp L, Lui YW, Montalban X, walking speed. Brain magnetic resonance imaging is acquired using standardized Naismith RT, Nicholas JA, Pellegrini F, acquisition sequences on Siemens 3T scanners. Quantitative measures of brain volume Rovira A, Schulze M, Tackenberg B, Tintore M, Tivarus ME, Ziemssen T and lesion load are obtained. Using a separate consent, the patients contribute DNA, and Rudick RA (2020) Harnessing RNA, and serum for future research. The clinicians retain complete autonomy in using Real-World Data to Inform MS PATHS data in patient care. A shared governance model ensures transparent data Decision-Making: Multiple Sclerosis Partners Advancing Technology and and sample access for research. Health Solutions (MS PATHS). Front. Neurol. 11:632. Results: As of August 5, 2019, MS PATHS enrolment included participants doi: 10.3389/fneur.2020.00632 (n = 16,568) with broad ranges of disease subtypes, duration, and severity. Overall, Frontiers in Neurology | www.frontiersin.org 1 August 2020 | Volume 11 | Article 632 Mowry et al. Learning Health System in MS 14,643 (88.4%) participants contributed data at one or more time points. The average patient contributed 15.6 person-months of follow-up (95% CI: 15.5–15.8); overall, 166,158 person-months of follow-up have been accumulated. Those with relapsing–remitting MS demonstrated more demographic heterogeneity than the participants in six randomized phase 3 MS treatment trials. Across sites, a significant variation was observed in the follow-up frequency and the patterns of disease-modifying therapy use. Conclusions: Through digital health technology, it is feasible to collect standardized, quantitative, and interpretable data from each patient in busy MS practices, facilitating the merger of research and patient care. This approach holds promise for data-driven clinical decisions and accelerated systematic learning. Keywords: learning health system, multiple sclerosis, MS PATHS, digital health technology, standardized brain magnetic resonance imaging INTRODUCTION (also known as evidence-generating medicine) seeks to produce better outcomes and research based on real-time data acquisition The multiple sclerosis (MS) treatment landscape has experienced (10). The tenets of the LHS concept include collection of a dramatic evolution over the past two decades; there standardized, meaningful data on every patient seeking care, are now 16 approved disease-modifying therapies (DMTs). increased engagement of patients in the process of care, use of Approaches to personalized medicine in MS, however, have quantitative data for clinical decision-making, and aggregation not kept pace. Defining disease prognosis, treatment outcomes, of data from populations for systematic learning (10). The LHS monitoring treatment response, and determining optimal represents a culture of continuous learning, feeding insights treatment sequencing remain variable and somewhat subjective back into care delivery to continuously reanalyze, revalidate, and in MS practice. Personalized medicine efforts have focused both improve outcomes. on identifying informative patient phenotypes (1) and effectively The current report describes the first LHS in MS, Multiple integrating and visualizing individual patient data (2). Sclerosis Partners Advancing Technology and Health Solutions The rigorous collection and analysis of real-world data may (MS PATHS). In MS PATHS, quantitative clinical and imaging accelerate the development of personalized medicine in MS and data are collected in a standardized manner for each patient as address some of these gaps (3–5). Opportunities and challenges part of routine care. Data are collected as part of the patient’s related to data pooling and data standardization in MS were clinical evaluation, while—with a patient’s permission—the data recognized by early pioneers, beginning decades ago (6, 7), and are pseudonymized and aggregated for systematic learning. recent progress has been summarized (8, 9). These efforts have The program leverages technology and patient engagement to ushered in an era of data standardization and pooling in an automate data collection and analysis, minimizing the burden on attempt to extend systematic learning beyond structured research the care system and providers. The specific goals of MS PATHS protocols to more representative real-world populations. For are to better understand the disease, identify the predictors of maximum impact, data should include standardized and therapeutic responses, define and measure outcomes during the quantitative clinical, radiologic, and biological phenotyping from course of care, and develop approaches to personalized medicine. a heterogeneous population representative of the diversity of We aim to demonstrate the feasibility of implementing the key patients with MS seen in everyday clinical practice. features of an LHS in MS by describing the patient population The learning health system (LHS) model, as proposed by enrolled in MS PATHS to date, initial data completion rates, the Institute of Medicine, outlines a method to enable broad- differences between MS PATHS and typical MS clinical trial scale quantitative patient phenotyping through the merging patients, heterogeneity of practice across the network, and of clinical research and healthcare delivery (4, 10). The LHS utilization of the research data. Abbreviations: DMT, disease-modifying therapy; EMR, electronic medical MATERIALS AND METHODS record; FLAIR, fluid-attenuated inversion recovery; IQR, interquartile range; IT, information technology; LHS, learning health system; MPRAGE, magnetization- Initial Design prepared rapid gradient-echo imaging; MRI, magnetic resonance imaging; MS, MS PATHS is a collaborative network of healthcare institutions multiple sclerosis; MS PATHS, Multiple Sclerosis Partners Advancing Technology that have standardized elements of their clinical assessments and and Health Solutions; MSPT, Multiple Sclerosis Performance Test; NA, not collaborated with Biogen to implement a centralized database applicable; Neuro-QoL, Quality of Life in Neurological Disorders; PPMS, primary progressive multiple sclerosis; PRMS, progressive relapsing multiple sclerosis; for research purposes. This network was designed based on the RCT, randomized controlled clinical trial; RRMS, relapsing-remitting multiple guiding principles described in Table 1. In 2014, a group of sclerosis; SPMS, secondary progressive multiple sclerosis. stakeholders from Cleveland Clinic, Johns Hopkins University, Frontiers in Neurology | www.frontiersin.org 2 August 2020 | Volume 11 | Article 632 Mowry et al. Learning Health System in MS TABLE 1 | Guiding principles for MS PATHS. collaboration with Siemens Healthineers, a software prototype is currently in development, with a focus on workflow Engage all healthcare providers and nearly all patients with MS in a healthcare institution integration and performance adequate for individual patient Standardize, quantify, and maximize data collected as part of standard of care clinical decision-making. • Identify key data points needed to interpret test results, achieve practice To enable translational research, an MS PATHS research standards/meaningful use, and enable the generation of new substudy was implemented to enable the collection of blood clinical knowledge samples under a research protocol that could be linked to the Leverage technology to enable data collection in clinical practice standardized clinical and imaging data for future analyses. • Make it possible to collect data on all participating patients, which is too time consuming using traditional research methods • Leverage patient-reported data to the greatest extent possible Participating Healthcare Institutions • Make MSPT data available at the point of care and simultaneously aggregate To be eligible for MS PATHS, the participating healthcare data for learning institutions had to have an MS center with an active census of Collect data outside of standard of care via separately consented substudies ≥500 patients that routinely used functional measures and MRI Ensure transparent governance by multi-stakeholder group in their clinical practice and a willingness to further standardize Become recognized as meaningful by patients, providers, payers, and other stakeholders aspects of their clinical and radiological assessments. In addition, each participating healthcare institution had to have at least MS, multiple sclerosis; MS PATHS, Multiple Sclerosis Partners Advancing Technology and one Siemens 3T MRI scanner available for clinical use. Each Health Solutions; MSPT, Multiple Sclerosis Performance Test. participating center had to be willing to implement a centralized health information exchange architecture to transfer the data to a research database and obtain approval of the project protocol by New York University, and Biogen began the planning process the institutional review board or ethics committee, information to best design the tools, systems, and governance needed for an technology (IT) security, and/or data privacy committee and LHS in MS. agree to adhere to Good Clinical Practice and ethical principles Methods for standardized clinical and imaging data collection as outlined in the Declaration of Helsinki. The participating that do not increase the burden on providers or generate healthcare institutions of each investigator receive financial significant incremental cost were an important aspect of the compensation for data shared and biosamples collected as part planning process. Technology was developed to allow patient of this program based on fair market value. self-administered clinical assessment, resulting in standardized, high-quality clinical data. Technician-based testing—specifically, Patients Multiple Sclerosis Functional Composite testing—was adapted to Patients with a confirmed diagnosis of MS, including clinically a series of patient self-administered iPad-based tests to provide isolated syndrome, and the ability to understand the purpose and quantitative data related to processing speed, low-contrast visual the risks of the project are eligible to enroll in MS PATHS. In acuity, manual dexterity, and walking speed (11). This was done contrast with a traditional prospective observational study, the in order to facilitate the neuroperformance testing for every sites in MS PATHS agree to adopt the outlined standard of care, patient with MS, even in a busy clinical practice (12–14). The and the patients provide authorization for the use of protected iPad-based clinical assessment tool, called the Multiple Sclerosis health information in accordance with national and local subject Performance Test (MSPT), also enabled the collection of a privacy regulations. Authorization format was determined by structured patient history. the local institutional review board or ethics committee and Because the clinical assessment was patient self-administered, ranges from the use of a standard medical information privacy there are differences between the MSPT and the physician- waiver (four institutions) to an oral consent (one institution) to derived measures used in clinical trials and traditional MS a full informed consent (five institutions). The investigators and practice registries. Neurologist-determined relapses and research coordinators are encouraged to invite all patients at each Expanded Disability Status Scale were replaced by patient- MS center to participate. At steady state, the network aims to have reported relapse and Patient Determined Disease Steps. Prior 80% of the MS patients at each participating MS center enrolled studies support the validity of self-reported relapses (15, 16) in MS PATHS. and showed a strong correlation between Patient Determined Disease Steps and Expanded Disability Status Scale (17–19). The Procedures MSPT also enabled the collection of quality-of-life data. The Upon enrollment, the patients are assigned a unique MS PATHS Quality of Life in Neurological Disorders (Neuro-QoL) measure identification number that acts as the patient identifier in the was selected as a standardized, well-validated patient-reported LHS. This allows linkage of pseudonymized data from different outcome instrument (20–22). The computer-adapted version sources for the same patient. The authorization form allows of Neuro-QoL was incorporated into the MSPT to minimize for prospective data sharing as well as sharing of data from 12 administration time. months before the date of consent. Two standardized magnetic resonance imaging (MRI) The data elements collected routinely for patients in MS acquisition sequences were incorporated into routine MS PATHS are listed in Table 2. Clinical data are collected using imaging protocols to enable the reliable calculation of the MSPT. In addition, structured clinical data are shared from brain atrophy and lesions at the point of care. Through a electronic medical records at each institution. Imaging data Frontiers in Neurology | www.frontiersin.org 3 August 2020 | Volume 11 | Article 632 Mowry et al. Learning Health System in MS TABLE 2 | Current data elements collected in the MS PATHS learning health TABLE 3 | Standardized Siemens 3T brain MRI sequence parameters for MS system. PATHS. Patient demographic information (MSPT and EMR) Parameter 3D FLAIR MS-Pie 3D T1 MS-Pie • Age (EMR) (SPACE) (MPRAGE) • Gender (MSPT) • Race (MSPT) Resolution (mm) 1×1×1 1×1×1 • Ethnicity (MSPT) Field of view 256 × 256 × 176 256 × 256 × 176 • Education (MSPT) Orientation Sagittal Sagittal • Employment status (MSPT) • Insurance coverage type (MSPT and EMR) Total acquisition time (min:s) 6:27 5:12 • Living situation (MSPT) Repetition time (ms) 5,000 2,300 • Employment status (MSPT) Echo time (ms) 392 2.96 • Dominant hand (MSPT) Inversion time (ms) 1,800 900 MS and other medical history (MSPT and EMR) 3D, three-dimensional; FLAIR, fluid-attenuated inversion recovery; MPRAGE, • MS subtype (MSPT) magnetization-prepared rapid gradient-echo imaging; MRI, magnetic resonance imaging; • Age at first MS symptom onset (MSPT) ms, milliseconds; MS PATHS, Multiple Sclerosis Partners Advancing Technology and • Age at MS diagnosis (MSPT) Health Solutions. • Mobility aid use (MSPT) • Smoking status (EMR) include brain MRIs acquired using two standardized sequences Physical and laboratory assessments (EMR) • Weight and height (three-dimensional fluid-attenuated inversion recovery and • Body mass index three-dimensional T1 acquired on Siemens 3T scanners; • Blood pressure Table 3), information from radiology reports, and quantitative • Laboratory test values measures of brain atrophy and lesion metrics derived from an image analysis software prototype. Medications (MSPT and EMR) • Patient self-reported use of MS disease-modifying therapy (MSPT) Data collected during patient care visits can also be linked • Medication list (EMR) to substudies. Each substudy requires a separate protocol and informed consent to allow for research assessments or procedures Patient-reported outcomes and tests (MSPT) and for the linkage of the substudy data with patient care data. • Patient-reported relapses An example is an ongoing biorepository substudy that collects • Patient Determined Disease Steps • Neuro-QoL a one-time blood sample for future genomic analyses (10 ml for ◦ Mental adult patients and 6 ml for pediatric patients) and repeated blood ‚Ability to participate in social roles and activities samples for future biomarker analyses (33 ml for adult patients ‚Anxiety and 13.5 ml for pediatric patients) not more frequently than every ‚Cognition 6 months. The clinical phenotyping information for any genetic ‚Depression ‚Emotional and behavioral dyscontrol or biomarker analyses will be derived from the routine clinical ‚Positive affect and well-being care data that are shared as part of MS PATHS. ‚Satisfaction with social roles and activities ‚Stigma Governance ◦ Physical The governance for MS PATHS requires a structure and a ‚Fatigue ‚Lower extremity function (mobility) process that: ‚Sleep disturbance (1) Are fair, transparent, and compliant for all ‚Upper extremity function (fine motor, ADL) • Neuroperformance testing participating organizations; ◦ Walking Speed Test (2) Assure confidence from all participating organizations; ◦ Processing speed test (3) Foster collaboration; ◦ Manual dexterity test (4) Reinforce the clinical and research integrity of participating ◦ Contrast sensitivity test healthcare institutions; MRI-related data (5) Ensure that all participating organizations have sufficient • 3D T1 and 3D FLAIR MRI freedom to operate; and • Radiologist report of number of new or enlarging T2 lesions (EMR) (6) Foster innovation, limit risk, and enable long-term success. • Radiologist report of number of enhancing T2 lesions (EMR) • Quantitative brain volume metrics* A steering committee was tasked with: • Quantitative T2 lesion metrics* (1) Providing strategic and operational guidance; 3D, three-dimensional; ADL, activities of daily living; EMR, electronic medical record; (2) Setting the MS PATHS scientific strategy; FLAIR, fluid-attenuated inversion recovery; MRI, magnetic resonance imaging; MS, (3) Creating and overseeing data and sample access rules; multiple sclerosis; MSPT, Multiple Sclerosis Performance Test; Neuro-QoL, Quality of Life in Neurological Disorders. (4) Monitoring performance; and *Under development. (5) Providing other oversight activities as needed. Frontiers in Neurology | www.frontiersin.org 4 August 2020 | Volume 11 | Article 632 Mowry et al. Learning Health System in MS The steering committee consists of six representatives from imaging and three-dimensional fluid-attenuated inversion participating healthcare institutions and one representative recovery) that are readily available product sequences and from Biogen. Three steering committee seats are reserved for consistent with recent MS imaging guidelines (24). The investigators from the institutions that collaborated with Biogen participating healthcare institutions have also implemented on the initial planning process (Cleveland Clinic Foundation, standardized fields in the radiology report for assessment of Johns Hopkins University, and New York University), and one new or enlarging T2 lesions and contrast-enhancing lesions seat each is designated for an additional US healthcare institution, (if applicable). an EU healthcare institution, and a radiologist in MS PATHS. The steering committee governs a set of data and sample access subcommittees responsible for approving requests for Health Information Exchange Architecture MS PATHS is enabled by health IT that supports the secure MS PATHS data and samples. The healthcare institutions transfer, processing, and harmonization of clinical data for and Biogen each have separate data and sample access research purposes. Similar to the hub-and-spoke model of data committees. Biogen employees do not sit on the healthcare transfer in health information exchanges, clinical source systems institution committees and vice versa. Separate subcommittees such as the electronic medical record systems at participating for Biogen and participating healthcare institutions were set up healthcare institutions (the spokes) will send data through a series to ensure the independence of research led by the participating of intermediary systems before the data are passed through to the healthcare institutions. The healthcare institution and Biogen central MS PATHS research database (also known as the LHS). subcommittees operate under the same procedures, and all Each participating institution is supported by a separate approved uses of the data or the samples are posted on the MS gateway that is responsible for applying consent logic to PATHS research website, which is accessible to Biogen and the incoming data and then conducting any needed format participating healthcare institutions to ensure transparency and transformations to enable ingestion by a central data broker. promote collaboration. In addition to approving data or sample The two data brokers, one for the United States and one for requests, these subcommittees are also responsible for reviewing the European Union, manage a patient registration and consent resulting presentations or publications to ensure alignment with index as well as a de-identification tool that pseudonymizes data. the original request. Biogen, as the sponsor, has no role in Patient data other than consent and registration information are writing or editing publications unless a Biogen employee is deleted from each broker after five business days. a co-author. Biogen has contracted with an IT vendor to act as a trusted third party to build and operate the MS PATHS Standardization gateways and data brokers. This IT vendor is responsible for Participating healthcare institutions have implemented the protecting and processing identifiable patient data before they are MSPT (11, 13, 14), an iPad-based medical assessment tool that pseudonymized and sent to the LHS. quantifies major MS-associated motor, visual, and cognitive In the United States, the vendor has entered into business symptoms and quality-of-life outcomes. The MSPT incorporates associate agreements with each participating healthcare a structured patient history (gathers the patient’s relevant institution. Biogen does not have access to the data in either demographic and socioeconomic information, MS history, MS broker. In the European Union, each healthcare institution has treatment information, and self-reported disability using the contracted with an additional IT vendor to serve as the initial Patient Determined Disease Steps) (17–19), 12 subscales of intermediary prior to the edge gateway. The EU IT vendor the Neuro-QoL (20–22), and an electronic adaptation of the completes an initial pseudonymization of the data before they Multiple Sclerosis Functional Composite (23). The adapted are transferred to an institution’s gateway and subsequently to Multiple Sclerosis Functional Composite includes a processing the EU broker. Once the data are pseudonymized in a broker, speed test that is similar to the Symbol Digit Modalities they are transferred to the LHS. The LHS is logically isolated Test; a manual dexterity test, similar to the 9-Hole Peg Test; from the brokers so that both Biogen and researchers from a contrast sensitivity test, similar to the Sloan low-contrast participating healthcare institutions do not have access to visual acuity test; and a 25-foot walking speed test, similar identifiable patient data. to the Timed 25-Foot Walk. The MSPT is administered In the LHS, data are harmonized using industry-accepted during routine clinical visits, typically prior to meeting with clinical terminology standards such as SNOMED, Logical the healthcare provider. Depending on the institution, the Observation Identifiers Names and Codes, RxNorm, National results are immediately available to the healthcare provider Drug Code, and International Statistical Classification of Diseases via the patient’s electronic medical record or via a results and Related Health Problems. The LHS data are available to screen on the MSPT. The processing speed test, manual be requested for research purposes by any researcher at a dexterity test, contrast sensitivity test, and 25-foot walking participating healthcare institution or at Biogen. speed test demonstrate reliability, validity, and sensitivity to MS outcomes (13, 14). In MS PATHS, the participating healthcare institutions Statistical Analysis collaborate with Biogen and Siemens Healthineers to implement Descriptive analyses were used to describe MS PATHS patient two highly standardized MRI acquisition sequences (a three- characteristics at the time of a patient’s initial MSPT assessment. dimensional magnetization-prepared rapid gradient-echo Continuous variables were reported as mean (SD) and categorical Frontiers in Neurology | www.frontiersin.org 5 August 2020 | Volume 11 | Article 632 Mowry et al. Learning Health System in MS TABLE 4 | Patient demographic and clinical characteristics at initial assessment. TABLE 4 | Continued Characteristic* Value Characteristic* Value Mean (SD) age (years)† 47.0 (12.4) Depression 47.3 (8.0) Female, n (%)‡ 10,712 (73.2) Emotional behavioral dyscontrol†† 50.6 (9.9) Race/area of origin, n (%)‡ Positive affect or well-being† † 52.7 (7.2) United States—race 11,236 (76.7) Satisfaction with social roles 47.0 (7.5) White 8,933 (79.5) Stigma 47.9 (8.5) Black or African American 1,419 (12.6) Physical Asian 75 (0.7) Fatigue 49.4 (9.9) American Indian or Alaska Native 46 (0.4) Lower extremity function 46.7 (11.2) Native Hawaiian or other Pacific Islander 10 (0.1) Sleep 52.7 (10.2) Multiple 290 (2.6) Upper extremity function 45.4 (9.6) Other/unknown 368 (3.3) DMT, disease-modifying therapy; MS, multiple sclerosis; Neuro-QoL, Quality of Life in Choose not to report 95 (0.9) Neurological Disorders. European Union—area of origin 3,407 (23.3) *Data are reported as mean (SD) or n (%) for continuous and categorical Western Europe 2,927 (85.9) variables, respectively. † n = 14,484, †† n = 11,567, ‡ n = 14,643, § n = 13,904, ¶ n = 14,236. Eastern Europe 197 (5.8) # Includes interferon beta-1a, interferon beta-1b, interferon beta-other, and Asia 12 (0.4) peginterferon beta-1a. Multiple 64 (1.9) **Mean (SD) score reference value 50 (10); n = 11,827. Other/unknown 138 (4.1) Choose not to report 69 (2.0) Age at diagnosis (years)§ 35.4 (11.2) variables as percentages. Between-group differences were Age at first symptoms (years)¶ 32.6 (11.4) assessed using t tests and chi-square (χ 2 ) tests, as appropriate. MS subtype, n (%)‡ To test whether MS PATHS is more diverse than typical Relapsing remitting 8,708 (59.5) phase 3 clinical trials in MS, the characteristics of patients with Secondary progressive 2,504 (17.1) relapsing–remitting MS (RRMS) enrolled in MS PATHS were Progressive relapsing 1,247 (8.5) compared with patients with RRMS pooled from six phase 3 Primary progressive 1,100 (7.5) randomized controlled clinical trials (RCTs) sponsored by Biogen Missing 1,084 (7.4) (25–29). Comparisons of patients from MS PATHS vs. the pooled Number of relapses in past 12 months, n (%)‡ RCTs were made using Wilcoxon rank-sum tests and Pearson 0 7,615 (52.0) χ 2 tests. Standardized mean or proportion differences (i.e., effect 1 3,156 (21.6) sizes) were also calculated. C statistics for membership (30) and 2 1,915 (13.1) predicted probability (i.e., propensity score) distributions were ≥3 1,705 (11.6) estimated using multivariable logistic regression models to assess Missing 252 (1.7) overall baseline characteristics (i.e., case–mix) similarity. To Baseline DMT use‡ assess the extent of overlap in the two populations (MS PATHS vs. Dimethyl fumarate 1,966 (13.4) RCTs), 1:1 propensity score matching (31) was performed based Glatiramer acetate 1,761 (12.0) on a 5:1 greedy match algorithm (32). Comparisons between the Fingolimod 1,645 (11.2) matched sample characteristics were made using McNemar’s test or Wilcoxon signed rank test. Effect sizes and C statistics for Interferon# 1,492 (10.2) membership were also reported. Natalizumab 1,384 (9.5) To assess heterogeneity in terms of assessment frequency, Ocrelizumab 748 (5.1) separate Kaplan–Meier plots were created for each center, Teriflunomide 562 (3.8) describing the time (months) between assessments. Rituximab 272 (1.9) Heterogeneity at the level of the healthcare institution was Alemtuzumab 230 (1.6) tested using log-rank tests. Separate analyses were conducted for Other 115 (0.8) MSPT and MRI assessments. Not taking any medication/medication not listed 4,418 (30.2) Missing 50 (0.3) Neuro-QoL T-score** RESULTS Mental Ability to participate in social roles 47.3 (8.0) MS PATHS Patient Population As of August 5, 2019, 16,568 patients from 10 participating Anxiety 51.1 (9.4) institutions in the United States (n = 7) and the European Union Cognitive functioning 46.2 (9.0) (n = 3) agreed to share their clinical data, representing 71.4% of (Continued) patient census seen in the MS clinics within the network. As of the Frontiers in Neurology | www.frontiersin.org 6 August 2020 | Volume 11 | Article 632 Mowry et al. Learning Health System in MS TABLE 5 | Neuroperformance scores at initial assessment. Mean (SD) parameter All RRMS SPMS PRMS PPMS F test, p value Processing speed test (number correct) 46.6 (13.1) 50.0 (12.2) 41.3 (11.8) 40.3 (13.1) 39.8 (12.6) F = 532.97 n = 13,250 n = 8,059 n = 2,188 n = 1,075 n = 966 p < 0.0001 Contrast sensitivity test (number correct) 34.2 (12.7) 36.3 (11.6) 29.9 (13.6) 30.8 (13.5) 29.6 (14.5) F = 140.74 n = 8,277 n = 5,254 n = 1,222 n = 611 n = 541 p < 0.0001 Manual dexterity test (seconds) 27.4 (6.8) 25.9 (5.9) 30.9 (7.4) 30.1 (7.6) 31.0 (7.7) F = 459.18 n = 11,829 n = 7,505 n = 1,834 n = 880 n = 758 p < 0.0001 Walking speed test (seconds) 7.5 (4.6) 6.6 (3.2) 9.7 (6.4) 8.8 (5.3) 10.1 (7.1) F = 362.83 n = 11,758 n = 7,504 n = 1,778 n = 899 n = 747 p < 0.0001 PPMS, primary progressive multiple sclerosis; PRMS, progressive relapsing multiple sclerosis; RRMS, relapsing-remitting multiple sclerosis; SPMS, secondary progressive multiple sclerosis. cutoff date for the data included in this manuscript, the number TABLE 6 | MSPT module completion rates at initial assessment. of withdrawals was 158 (0.95%) and ranged from 0.12 to 2.91% of participants across the sites. Module Assigned, Completed, Patient declined or Of 16,568 enrolled patients, 14,643 patients completed at least n* n (%) unable to complete, one MSPT assessment; the characteristics at the time of initial n (%) MSPT assessment for these patients are shown in Table 4. Mean Processing speed test 14,643 13,250 (90.5) 1,393 (9.5) (SD) age was 47.0 (12.4) years and the population was largely Contrast sensitivity test 14,643 8,277 (56.5) 6,366 (43.5) female [n = 10,712 (73.2%)] and predominantly white, although Manual dexterity test 14,643 11,829 (80.8) 2,814 (19.2) the absolute number of non-white participants is substantial. Walking speed test 14,643 12,152 (83.0) 2,491 (17.0) Mean (SD) Neuro-QoL T-scores ranged from 45.4 (9.6) to † Neuro-QoL 14,643 11,827 (80.8) 2,816 (19.2) 52.7 (10.2). MyHealth‡ 14,643 14,643 (100.0) At their initial MSPT assessment, 69.6% of 14,643 patients reported the use of a DMT. The most frequently reported DMTs MSPT, Multiple Sclerosis Performance Test; Neuro-QoL, Quality of Life in Neurological were dimethyl fumarate, glatiramer acetate, and fingolimod. Disorders. * MSPT assessments taken on the same day are combined into a single record. † A computer-adaptive quality-of-life measure included with the MSPT. DMT use was highest in patients with RRMS [6,495 (74.6%)] ‡ A structured patient questionnaire that records demographics, health history, use of compared with patients with progressive MS [1,662 (66.4%) multiple sclerosis disease-modifying therapy, and multiple sclerosis status. secondary progressive, 540 (49.1%) primary progressive, and 806 (64.6%) progressive relapsing; χ 2 = 349.5; p < 0.0001]. The neuroperformance scores at initial assessment are patients with at least one MSPT assessment, 7,622 (52.1%) had summarized in Table 5. Patients with RRMS performed better at least one standardized 3T MRI. on neuroperformance testing compared with patients with A one-time genetic sample was collected from 6,320 patients. progressive disease. Other blood samples have been collected at a total of 10,223 biobanking visits from 6,581 unique patients, including 2,584/6,581 (39.3%) with a longitudinal sample collected Assessment Completion Rates and Data (Table 7). The 10,223 biobanking visits represent 129,986 Volume individual samples received in the central laboratory. A total The initial completion rates for the MSPT component modules of 265 (0.2%) samples were rejected after the initial quality ranged from 56.5% (8,277/14,643; contrast sensitivity test) to control checks. 100.0% (14,643/14,643; MyHealth module; Table 6). Missing data resulted from the coordinator disabling the module for patients Heterogeneity Observed Across Sites unable or unwilling to complete the test or the patient canceling After completing their initial MSPT, 64% and 81% of patients the test themselves. Longitudinal MSPT data were available in MS PATHS completed a follow-up MSPT within 12 and for 72.7% (10,640/14,643) of patients (Table 7). The average 24 months, respectively (Figure 1A). The median time to patient contributed 15.6 person-months of follow-up (95% CI: complete a follow-up MSPT was ∼7.4 months after the initial 15.5–15.8); overall, 166,158 person-months of follow-up have MSPT. The median time to complete a follow-up MSPT been accumulated. varied considerably among the participating healthcare A total of 14,414 MRI studies were collected from 8,364 institutions, ranging from 3 to 13 months (Figure 1B; unique patients, including 3,822/8,364 (45.7%) patients with χ 2 = 2,575; p < 0.0001). longitudinal MRI data (Table 7). Of the 14,414 MRI studies After completing their initial standardized 3T brain MRI, received, 281 (2.0%) were rejected for being incomplete or not 30% and 58% of patients in MS PATHS completed a follow-up acquired using standardized sequence parameters. Of 14,643 standardized 3T brain MRI within 12 and 24 months, respectively Frontiers in Neurology | www.frontiersin.org 7 August 2020 | Volume 11 | Article 632 Mowry et al. Learning Health System in MS TABLE 7 | MS PATHS data volume (N = 16,568). Assessment Total assessments ≥1, n (%) ≥2, n (%) ≥3, n (%) ≥4, n (%) MSPT 41,187 14,643 (100) 10,640 (72.7) 7,038 (48.1) 4,332 (29.6) Median (IQR) duration since first assessment (days) 448 (282–650) 553 (390–725) 609 (478–763) Brain MRI 14,414 8,364 (100) 3,822 (45.7) 1,510 (18.1) 470 (5.6) Median (IQR) duration since first assessment (days) 469 (350–715) 721 (565–834) 859 (721–1004) Biobanking 10,223 6,581 (100) 2,584 (39.3) 836 (12.7) 196 (3.0) Median (IQR) duration since first assessment (days) 364 (224–483) 546 (421–637) 658 (609–736) IQR, interquartile range; MRI, magnetic resonance imaging; MS PATHS, Multiple Sclerosis Partners Advancing Technology and Health Solutions; MSPT, Multiple Sclerosis Performance Test. (Figure 2A). The median time between the two standardized 3T There were three initial observations from MS PATHS. brain MRIs was ∼18 months. The median time to complete a The first relates to the characteristics of patients with MS. follow-up standardized 3T brain MRI varied considerably among Although the patients in MS PATHS demonstrate demographic the participating healthcare institutions, ranging from 12 to 24 and disease characteristics typical of other large MS cohorts, months (Figure 2B; n = 721; p < 0.0001). In four institutions, the clinical characteristics from patients with RRMS in MS half of the patients had not yet had a second standardized 3T PATHS are partially non-overlapping with the patients enrolled brain MRI. in clinical trials (25–29), thus reflecting a broader population. DMT use varied across the participating sites, in terms of both The patients in MS PATHS exhibited a broader range of ages, whether or not a patient was treated with a DMT (χ 2 = 163.3; disability levels, disease duration, and DMT use, presumably p < 0.0001) and which DMT was prescribed (χ 2 = 2,584.6; p < because MS PATHS aims to enroll all patients with a 0.0001; Table 8). diagnosis of MS, including clinically isolated syndrome, with no additional inclusion or exclusion criteria (e.g., requiring Heterogeneity Relative to Phase 3 MS RCTs a relapse in the past 12 months, as is typically required The patients in MS PATHS were significantly older and had in pivotal RCTs). MS PATHS is more racially diverse than a longer disease duration than the patients from the pooled previously reported trials; white patients make up 79.5% of the RCTs (Table 9). In addition, the proportion of patients with no US MS PATHS population in comparison with 93.7% of the relapses in the last 12 months was higher in MS PATHS (54.4%) oral DMT clinical trial study population (36). The minimal than in the pooled RCTs (1.4%). The logistic regression models inclusion criteria and the robust enrollment rates suggest predicting membership in MS PATHS resulted in a C statistic of that studies using the MS PATHS population will generate 0.91 vs. the pooled RCTs. A C statistic of 1.0 indicates no overlap; data that will be more generalizable than data from clinical 0.91 indicates that RCT patients were substantially distinct from trials, in which enrollment is more selective. It is also likely patients in MS PATHS. Propensity score 1:1 matching of the 8,708 that patient characteristics obtained from MS PATHS will be patients with RRMS in MS PATHS and 6,574 patients with RRMS more representative of the broader MS population than data in the pooled RCTs yielded 1,922 patients from each sample that from registries where a smaller subset of patients from a were able to be matched (Table 9). clinical practice in one location is selected for inclusion in the cohort (37). The second observation is the demonstrated feasibility Research Utilization of LHS Data of real-time quantitative patient phenotyping as part of The governing subcommittee has approved 58 data use requests clinical practice. As of August 5, 2019, 88.4% of patients and one sample use request. Based on these requests, 78 who gave permission for their clinical data to be used for conference abstracts and manuscripts have been published. Two research have completed at least one self-administered National Institutes of Health grants have been funded. neuroperformance assessment. Also, standardized MRI acquisition sequences were incorporated into clinical brain DISCUSSION MRI protocols, with 98% of scans passing the quality control assessments. The Institute of Medicine, a member of the US National The third observation relates to the variability in practice Academies, recommended foundational elements for an LHS patterns observed across participating healthcare institutions. (10) as an approach to address recognized challenges, including We observed a significant inter-institution variability in the the need for consistent quality and efficiency, in the US healthcare rate of return for follow-up and the average interval between system. These challenges are also recognized by healthcare assessments captured in MS PATHS, indicating that there is no systems and regulatory bodies outside of the United States such uniform standard for visit frequency in the network. Follow-up as the European Medicines Agency (2, 4, 33–35). As the first interval is important when determining outcomes because visit LHS in MS, MS PATHS was designed to incorporate each of the frequency may influence observed event frequency, depending foundational elements outlined in the introduction. on how the information is captured. For example, individuals Frontiers in Neurology | www.frontiersin.org 8 August 2020 | Volume 11 | Article 632 Mowry et al. Learning Health System in MS FIGURE 1 | Probability of completing a follow-up Multiple Sclerosis Performance Test (MSPT; as a function of months between the initial MSPT and completing the second MSPT), (A) overall and by (B) Multiple Sclerosis Partners Advancing Technology and Health Solutions center. with MS may not always present to the physician during (1) Creating trust in an academic–industry collaboration: The a relapse (38). initial collaborators were excited about the vision of MS PATHS; MS PATHS has experienced challenges and learnings during however, there were questions on how to build trust in a multi- the initial implementation, including: party collaboration with a biopharmaceutical company. The Frontiers in Neurology | www.frontiersin.org 9 August 2020 | Volume 11 | Article 632 Mowry et al. Learning Health System in MS FIGURE 2 | Probability of completing a follow-up standardized magnetic resonance imaging on a Siemens 3T scanner, (A) overall and (B) by individual Multiple Sclerosis Partners Advancing Technology and Health Solutions center. initial collaborators recognized that a transparent governance to improve outcomes for patients with MS as well as the guiding process would be key to engender trust among the participants principles (Table 1). The governance model was then developed and in the scientific community at large. All parties quickly over the course of six all-day meetings to enable all parties to aligned on a vision of using the LHS concept to generate evidence fulfill the shared vision and operate by the guiding principles. Frontiers in Neurology | www.frontiersin.org 10 August 2020 | Volume 11 | Article 632 Mowry et al. Learning Health System in MS TABLE 8 | Distribution of DMT use by participating healthcare institutions at initial assessment* . Institution DMT, n (%) Overall A B C D E F G H I J (n = 14,643) (n = 1,581) (n = 1,061) (n = 572) (n = 1,722) (n = 3,690) (n = 777) (n = 1,833) (n = 1,607) (n = 621) (n = 1,179) Alemtuzumab 230 (1.6) 12 (0.1) 3 (0.0) 4 (0.0) 103 (0.7) 19 (0.1) 14 (0.1) 4 (0.0) 28 (0.2) 0 (0.0) 43 (0.3) Azathioprine 6 (0.0) 0 (0.0) 0 (0.0) 0 (0.0) 0 (0.0) 0 (0.0) 0 (0.0) 0 (0.0) 2 (0.0) 2 (0.0) 2 (0.0) Cyclophosphamide 2 (0.0) 0 (0.0) 1 (0.0) 0 (0.0) 0 (0.0) 1 (0.0) 0 (0.0) 0 (0.0) 0 (0.0) 0 (0.0) 0 (0.0) Daclizumab 37 (0.3) 1 (0.0) 0 (0.0) 0 (0.0) 0 (0.0) 2 (0.0) 0 (0.0) 4 (0.0) 0 (0.0) 6 (0.0) 24 (0.2) Dimethyl fumarate 1,966 (13.4) 152 (1.0) 159 (1.1) 67 (0.5) 179 (1.2) 641 (4.4) 104 (0.7) 317 (2.2) 115 (0.8) 104 (0.7) 128 (0.9) Fingolimod 1,645 (11.2) 148 (1.0) 138 (0.9) 70 (0.5) 241 (1.6) 484 (3.3) 74 (0.5) 62 (0.4) 101 (0.7) 105 (0.7) 222 (1.5) Glatiramer acetate 1,760 (12) 324 (2.2) 63 (0.4) 88 (0.6) 159 (1.1) 395 (2.7) 95 (0.6) 330 (2.3) 158 (1.1) 39 (0.3) 109 (0.7) Immunoglobulin G 12 (0.1) 0 (0.0) 3 (0.0) 0 (0.0) 2 (0.0) 6 (0.0) 1 (0.0) 0 (0.0) 0 (0.0) 0 (0.0) 0 (0.0) Interferon beta-1a 1,088 (7.4) 171 (1.2) 50 (0.3) 44 (0.3) 67 (0.5) 295 (2.0) 41 (0.3) 129 (0.9) 225 (1.5) 31 (0.2) 35 (0.2) Interferon beta-1b 245 (1.7) 12 (0.1) 10 (0.1) 11 (0.1) 13 (0.1) 13 (0.1) 9 (0.1) 14 (0.1) 130 (0.9) 16 (0.1) 17 (0.1) Interferon beta-other 16 (0.1) 2 (0.0) 4 (0.0) 0 (0.0) 2 (0.0) 6 (0.0) 0 (0.0) 2 (0.0) 0 (0.0) 0 (0.0) 0 (0.0) Methotrexate 27 (0.2) 1 (0.0) 2 (0.0) 0 (0.0) 0 (0.0) 21 (0.1) 0 (0.0) 2 (0.0) 0 (0.0) 1 (0.0) 0 (0.0) Mitoxantrone 6 (0.0) 0 (0.0) 1 (0.0) 0 (0.0) 2 (0.0) 1 (0.0) 0 (0.0) 0 (0.0) 0 (0.0) 1 (0.0) 1 (0.0) Mycophenolate mofetil 24 (0.2) 3 (0.0) 1 (0.0) 0 (0.0) 2 (0.0) 13 (0.1) 0 (0.0) 5 (0.0) 0 (0.0) 0 (0.0) 0 (0.0) Natalizumab 1,384 (9.5) 139 (0.9) 155 (1.1) 53 (0.4) 271 (1.9) 263 (1.8) 65 (0.4) 217 (1.5) 95 (0.6) 34 (0.2) 92 (0.6) Ocrelizumab 748 (5.1) 27 (0.2) 60 (0.4) 64 (0.4) 140 (1.0) 173 (1.2) 21 (0.1) 108 (0.7) 23 (0.2) 27 (0.2) 105 (0.7) Ofatumumab 1 (0.0) 0 (0.0) 0 (0.0) 0 (0.0) 1 (0.0) 0 (0.0) 0 (0.0) 0 (0.0) 0 (0.0) 0 (0.0) 0 (0.0) Peginterferon beta-1a 159 (1.1) 14 (0.1) 15 (0.1) 17 (0.1) 5 (0.0) 26 (0.2) 5 (0.0) 38 (0.3) 22 (0.2) 7 (0.0) 10 (0.1) Rituximab 272 (1.9) 45 (0.3) 67 (0.5) 18 (0.1) 18 (0.1) 15 (0.1) 7 (0.0) 65 (0.4) 31 (0.2) 3 (0.0) 3 (0.0) Teriflunomide 562 (3.8) 54 (0.4) 39 (0.3) 41 (0.3) 85 (0.6) 76 (0.5) 39 (0.3) 42 (0.3) 104 (0.7) 28 (0.2) 54 (0.4) Other 1,077 (7.4) 139 (0.9) 74 (0.5) 20 (0.1) 110 (0.8) 298 (2.0) 68 (0.5) 98 (0.7) 103 (0.7) 58 (0.4) 109 (0.7) None 3,341 (22.8) 337 (2.3) 214 (1.5) 75 (0.5) 317 (2.2) 922 (6.3) 234 (1.6) 395 (2.7) 465 (3.2) 159 (1.1) 223 (1.5) Missing 35 (0.2) 0 (0.0) 2 (0.0) 0 (0.0) 5 (0.0) 20 (0.1) 0 (0.0) 1 (0.0) 5 (0.0) 0 (0.0) 2 (0.0) DMT, disease-modifying therapy. *Likelihood ratio (DMT × site) χ 2 = 2,584.6; p < 0.0001. (2) Aligning on which data to collect and how to standardize technical challenges. We started with a basic design of a health data collection: Given the broad research goals of MS PATHS, information exchange platform and iterated after receiving the potential data to collect were vast. In the initial planning feedback from key IT stakeholders at the founding institutions. discussions, it became clear that standardizing and collecting Because IT architecture is different in each medical center, all potential data would not fit into the clinical workflow. interfaces for the IT platforms need to be customized for Data were prioritized in terms of time and utility. The each medical center to ensure compatibility and adherence to goal was to keep MSPT administration no longer than security requirements. The current implementation maintains 30 minutes for the average patient with MS, keep the brain consistent data handling and processing across the network MRI total scan time unchanged, and add no additional time but flexes to allow for variability in the interfaces and whether to a clinician’s clinical documentation. Any suggested patient- a push or pull model is optimal for each individual medical reported question or scale was weighed in terms of time center. This type of model was aided by the increasing adoption to complete and its potential utility for clinical care and of common healthcare data standards across healthcare research. Implementing axial reconstruction of the standardized in general. three-dimensional sequences allowed radiologists/neurologists (4) Ensuring compliance with US and EU data privacy to view scans as they normally would without the need for regulations: MS PATHS was designed to be both HIPAA and duplicative MRI acquisitions. Also, collaborating with Siemens GDPR compliant. In MS PATHS, patient data are only shared Healthineers allowed the sequences to be optimized based on for research purposes, with patient consent. Trusted third parties feedback from MS PATHS radiologists to enable broad adoption serve as intermediaries to remove patient identifiers before the in the network. data are aggregated for research under appropriate contractual (3) Implementation of common health IT platforms: arrangements. Garnering acceptance and approval from a myriad Generating data in real time for clinical decision-making and of stakeholders at each institution is equally as important as providing pseudonymized data for research involved major a sound program design. MS PATHS has shown that with a Frontiers in Neurology | www.frontiersin.org 11 August 2020 | Volume 11 | Article 632 Mowry et al. Learning Health System in MS TABLE 9 | Demographic and clinical characteristics of patients with RRMS in MS PATHS and Biogen phase 3 RCTs (25–29). Unmatched population Propensity score-matched population Variable* MS PATHS RCTs Cohen’s MS PATHS RCTs Cohen’s d value d value N 8,708 6,574 1,922 1,922 Age (years) 45.0 (12.2) 37.4 (8.9) −0.71 40.7 (11.3) 40.3 (8.7) −0.04 Male (%) 23.9 28.3 0.10 25.4 26.4 0.02 Body mass index (kg/m2 ) 28.9 (7.0) 25.1 (5.4) −0.60 27.9 (6.3) 27.8 (6.8) −0.01 MS duration (years) 11.3 (9.0) 4.8 (5.0) −0.90 7.2 (6.3) 7.1 (6.3) −0.01 Manual dexterity test/9-Hole Peg 25.9 (5.9) 22.1 (6.1) −0.63 25.1 (4.7) 25.3 (8.0) 0.03 Test (seconds)† 25-foot walking speed test/Timed 6.6 (3.2) 6.4 (4.5) −0.04 6.5 (3.0) 6.5 (3.3) 0.01 25-Foot Walk (seconds)† Number of relapses in the past 12 months NA 0.8 4.6 0.24 1.9 2 0.01 0 54.4 1.4 −1.46 4.8 4.8 0 1 23.8 58.8 0.76 54.9 54.1 −0.02 2 12.6 29 0.41 26.5 26.7 0 ≥3 8.4 6.1 −0.09 11.8 12.4 0.02 MS, multiple sclerosis; MS PATHS, Multiple Sclerosis Partners Advancing Technology and Health Solutions; NA, not applicable; RCT, randomized clinical trial; RRMS, relapsing-remitting multiple sclerosis. *Data are reported as mean (SD) or n (%) for continuous and categorical variables, respectively. Cohen’s d values represent standardized mean or proportion differences (i.e., effect sizes). An absolute value of % d >10 is considered as clinically meaningful. † The manual dexterity test and 25-foot walking speed test were administered in MS PATHS and the 9-Hole Peg Test and Timed 25-Foot Walk were administered in the RCTs. sound program design, stakeholder engagement, and patient through conference calls, investigator meetings, and site visits. permission, data aggregation is feasible to conduct under both For example, an early MSPT implementation insight shared HIPAA and GDPR. across the network was that patients generally have a better (5) Enabling efficient data sharing: Our chosen hub-and- testing experience if they take the test without staff, a family spoke data sharing model simplified contractual relationships in member, or a caregiver by their side. MS PATHS. Each healthcare institution negotiated directly with (7) Real-life experience highlighted gaps in original technical Biogen rather than having to enter into a multi-lateral negotiation assumptions: Some design issues that have emerged and are for the initial contract and any subsequent amendments. All being corrected include issues with MSPT functionality (e.g., stakeholders acknowledged the logistical benefits of this model. lower-than-expected contrast sensitivity test completion rates) However, it required key clauses on the scope of data sharing, and issues with optimizing data collection methods for certain data permissions, and intellectual property to be uniform variables (e.g., comorbidities, medication start/stop dates). across all contracts and informed consent forms and for all Among the advantages of an academic and industry partnership parties to be comfortable that we could not accommodate model are the real-time and continuous feedback, re-assessment, one-off deviations if we were to maintain an effective and refinement of data gathering and overall strategy that hub-and-spoke model. facilitate continuous improvement. One example of this feedback (6) Re-engineering of clinical workflows involved all aspects loop is the creation of a MS SmartForm by one of the MS PATHS of clinic operations: One example of how MS clinics needed institutions that is freely available in EPIC foundation to all to change their operations was the incorporation of the MSPT EPIC users. This SmartForm will facilitate the standardization of into the clinical workflow. This necessitated designating space in key MS-related variables such as relapses and disease-modifying the clinic as a MSPT testing area, training staff on the MSPT, therapy start/stop dates to primarily aid clinical care, but with introducing MS patients to a new aspect of their visit, and the secondary benefit of improving data quality for research. adjusting patient arrival times whether formally through a new The US MS PATHS centers are currently in various stages of appointment time or reminders to arrive 30 min before their SmartForm implementation, while we are coordinating with the scheduled visit with their healthcare provider. The workflow EU MS PATHS centers to harmonize as much as possible. challenges differed somewhat for implementing the MSPT, (8) Incorporating the voice of the patient: Patients have always implementing the standardized MRI protocol, and research been identified as a key stakeholder in MS PATHS. During the workflows such as biobanking. Enthusiasm for the LHS tenets design phase, the investigators engaged a local patient advisory and the project sustained commitment from academic and group, when available, on the MS PATHS concept. The MSPT was industry project leaders, overcoming a myriad of challenges. The designed to facilitate the voice of the patient being incorporated network investigators also committed to sharing best practices into the clinical visit through the neuroperformance modules Frontiers in Neurology | www.frontiersin.org 12 August 2020 | Volume 11 | Article 632 Mowry et al. Learning Health System in MS and 12 domains of quality-of-life as assessed by the Neuro-QoL. particular, incorporating point-of-care quantitative MRI-based Prior to the initial deployment of the MSPT in MS PATHS, metrics in clinical practice has the potential to transform qualitative feedback was obtained by the patients in an initial precision medicine to the same degree that MRI-based metrics usability study (11). However, there is still more to be done to have advanced clinical trials. engage patients and complete the feedback loop as envisioned in Another important direction for future work in MS PATHS a LHS (10). To start, the network is working to generate regular is defining individual patient outcomes using quantitative patient newsletters that provide updates on MS PATHS and standardized data. As MS PATHS data accumulate, it may be summarize recent research presented at conferences or published possible to define complete MS disease control or actionable in journals. Each MS center will provide these updates via their change based on quantitative monitoring. Further, in the future, routine patient communication channels. We are also working decision support based on advanced analytic methods could on an updated publicly facing website that would also have help neurologists care for individuals with MS in the context of this information. a large number of DMTs and other therapeutic choices. Data (9) Moving from clinical implementation to clinical decision- accumulated in MS PATHS could support the development of making: Our efforts to date have been focused on the initial predictive models and decision support systems. implementation and data collection as highlighted in this paper. The LHS concept, exemplified by MS PATHS, could accelerate The network is now shifting focus to evidence generation that translational research in MS as evidenced by the number of active will enable the data collected in MS PATHS to be used more proposals and grant approvals. Tens of thousands of patients may routinely for clinical decision-making. Key next topics include be required to unravel the biological basis for MS heterogeneity understanding clinical cutoffs for the MSPT and MRI metrics, and develop personalized medicine based on individual patient looking at practice heterogeneity across the network to identify characteristics. Using quantitative data derived from clinical, best practices, and exploring feedback loops for investigators to imaging, and laboratory assessments generated during patient more easily use the data for quality improvement. care could drastically reduce the cost of, and thus make feasible, Despite the promise of this research, there are limitations to be such translational research studies. considered. Data missingness may be non-random. For example, patients with more disability may be less likely to have an MRI or complete the MSPT. The participating centers are referral DATA AVAILABILITY STATEMENT centers and may not fully represent broader MS populations. For example, the rates of patients with progressive disease on The datasets presented in this article are not readily available a DMT may be higher than expected, and incorporation of because MS PATHS is an ongoing study. Requests to access the the SmartForm data may provide additional insight. Still the datasets should be directed to datasharing@biogen.com. MS PATHS population represents an improvement over the populations included in clinical trials as it is more racially and clinically diverse. ETHICS STATEMENT Another limitation is related to the restriction of standardized The studies involving human participants were reviewed and MRI studies to Siemens 3T scanners. This was intentional in approved by Ethics committees or institutional review boards at order to remove scanner and manufacturer variability from the the following participating centers: Cleveland Clinic, Cleveland, derived metrics as the initial step in delivering point-of-care OH, USA; Johns Hopkins University, Baltimore, MD, USA; New MRI-based metrics to the clinician. As such, the time to follow-up York University, New York, NY, USA; OhioHealth, Columbus, MRI does not take into account the MRI exams acquired within OH, USA; Washington University in St. Louis, St. Louis, MO, each institution on non-Siemens 3T MRI scanners or using USA; University of Rochester Medical Center, Rochester, NY, different (non-standardized) acquisition sequences or an MRI USA; Cleveland Clinic Lou Ruvo Center for Brain Health, obtained outside the healthcare institution. Therefore, differences Las Vegas, NV, USA; University Hospital of Giessen and in MRI follow-up time between centers may reflect variations in Marburg, Marburg, Germany; Vall d’Hebron University Hospital, the use of Siemens 3T scanners and variation in the percentage Barcelona, Spain; and University Hospital Carl Gustav Carus, of patients who obtain MRI exams outside of the healthcare Dresden, German. The patients/participants provided their institution. A few participating centers image their patients with written informed consent to participate in this study. MS exclusively on Siemens 3T scanners, while others use a wide variety of scanner types across internal and external imaging facilities. Translation of the imaging methods to non-Siemens 3T AUTHOR CONTRIBUTIONS scanners is planned. The initial implementation of MS PATHS has been successful JW, CM, EF, and RR conceived and designed the data plan. CM overall, although we are continuously optimizing aspects of and FP conducted the analyses. EM, RB, JW, and RR led the our operations. We anticipate its value to grow with time as writing of the manuscript. RR had final responsibility for the large-scale, longitudinal, and multidimensional quantitative data decision to submit for publication. All authors provided critical accumulate. These data will potentially drive new insights and aid feedback and helped shape the research, analysis, manuscript, in the development and the validation of new technologies. In had full access to the study data for interpretation and drafting of Frontiers in Neurology | www.frontiersin.org 13 August 2020 | Volume 11 | Article 632 Mowry et al. Learning Health System in MS the report, and contributed to the interpretation of the findings. in conducting any scientific or editorial review of resultant Editorial support was provided to authors and funded by the publications. Biogen also does not permit its commercial study funder. All decisions relating to manuscript writing and employees to access the data. content were made jointly by the authors. ACKNOWLEDGMENTS FUNDING This study was funded by Biogen (Cambridge, MA, USA). Biogen provides funding for MS PATHS. As the sponsor of MS Biogen provided funding for medical writing and editorial PATHS, Biogen facilitated the planning process and funded the support in the development of this manuscript. Linda Wagner development and ongoing operation/support of the MSPT and (Excel Scientific Solutions, Fairfield, CT, USA) wrote the first health information exchange architecture. Biogen entered into a draft of the manuscript based on input from the authors, research agreement with Siemens Healthineers to co-develop the and Elizabeth Wassmer (Excel Scientific Solutions, Fairfield, MRI metrics prototype. Biogen also compensates participating CT, USA) copyedited and styled the manuscript per journal healthcare institutions for data and samples contributed to MS requirements. Biogen reviewed and provided feedback on the PATHS at fair market value rates. Biogen serves as the hub for paper to the authors. The authors had full editorial control of contracting and data sharing. All the participating healthcare the paper and provided their final approval of all content. The institutions contracted with Biogen to receive consented patient authors thank all of the patients who are participating in the MS data and allow Biogen to share the data with other researchers PATHS initiative. for uses consistent with the informed consent. As the data aggregator, Biogen curates the data and samples and then makes SUPPLEMENTARY MATERIAL them available to researchers in the network who have an approved data or sample use proposal. To maintain the scientific The Supplementary Material for this article can be found integrity of the network, Biogen plays no role in adjudicating online at: https://www.frontiersin.org/articles/10.3389/fneur. data and sample requests from academic collaborators or 2020.00632/full#supplementary-material REFERENCES 12. Rudick RA, Miller D, Bethoux F, Rao SM, Lee JC, Stough D, et al. The Multiple Sclerosis Performance Test (MSPT): an iPad-based disability assessment tool. 1. Rotstein D, Montalban X. Reaching an evidence-based prognosis for J Vis Exp. (2014) 88:e51318. doi: 10.3791/51318 personalized treatment of multiple sclerosis. Nat Rev Neurol. (2019) 15:287– 13. Rao SM, Losinski G, Mourany L, Schindler D, Mamone B, Reece C, et al. 300. doi: 10.1038/s41582-019-0170-8 Processing speed test: validation of a self-administered, iPad R -based tool 2. Gourraud PA, Henry RG, Cree BA, Crane JC, Lizee A, Olson MP, et al. for screening cognitive dysfunction in a clinic setting. Mult Scler. (2017) Precision medicine in chronic disease management: the multiple sclerosis 23:1929–37. doi: 10.1177/1352458516688955 BioScreen. Ann Neurol. (2014) 76:633–42. doi: 10.1002/ana.24282 14. Rao SM, Galioto R, Sokolowski M, McGinley M, Freiburger J, 3. Frieden TR. Evidence for health decision making — beyond Weber M, et al. Multiple Sclerosis Performance Test: validation of randomized, controlled trials. N Engl J Med. (2017) 377:465– self-administered neuroperformance modules. Eur J Neurol. (2020) 75. doi: 10.1056/NEJMra1614394 27:878–86. doi: 10.1111/ene.14162 4. Sherman RE, Anderson SA, Dal Pan GJ, Gray GW, Gross T, Hunter NL, et al. 15. Nickerson M, Cofield SS, Tyry T, Salter AR, Cutter GR, Marrie RA. Impact of Real-world evidence — what is it and what can it tell us? N Engl J Med. (2016) multiple sclerosis relapse: the NARCOMS participant perspective. Mult Scler 375:2293–7. doi: 10.1056/NEJMsb1609216 Relat Disord. (2015) 4:234–40. doi: 10.1016/j.msard.2015.03.005 5. Ziemssen T, Hillert J, Butzkueven H. The importance of collecting 16. Nickerson M, Marrie RA. The multiple sclerosis relapse experience: structured clinical information on multiple sclerosis. BMC Med. (2016) patient-reported outcomes from the North American Research Committee 14:81. doi: 10.1186/s12916-016-0627-1 on Multiple Sclerosis (NARCOMS) registry. BMC Neurol. (2013) 6. Confavreux C. Establishment and use of multiple sclerosis registers–EDMUS. 13:119. doi: 10.1186/1471-2377-13-119 Ann Neurol. (1994) 36:S136–9. doi: 10.1002/ana.410360733 17. Hohol MJ, Orav EJ, Weiner HL. Disease steps in multiple sclerosis: a 7. Paty D, Studney D, Redekop K, Lublin F. MS COSTAR: a computerized longitudinal study comparing disease steps and EDSS to evaluate disease patient record adapted for clinical research purposes. Ann Neurol. (1994) progression. Mult Scler. (1999) 5:349–54. doi: 10.1177/135245859900500508 36:S134–5. doi: 10.1002/ana.410360732 18. Kister I, Chamot E, Salter AR, Cutter GR, Bacon TE, Herbert J. Disability in 8. Cohen JA, Trojano M, Mowry EM, Uitdehaag BM, Reingold SC, multiple sclerosis: a reference for patients and clinicians. Neurology. (2013) Marrie RA. Leveraging real-world data to investigate multiple sclerosis 80:1018–24. doi: 10.1212/WNL.0b013e3182872855 disease behavior, prognosis, and treatment. Mult Scler. (2020) 26:23– 19. Learmonth YC, Motl RW, Sandroff BM, Pula JH, Cadavid D. Validation 37. doi: 10.1177/1352458519892555 of Patient Determined Disease Steps (PDDS) scale scores in persons with 9. Flachenecker P, Buckow K, Pugliatti M, Kes VB, Battaglia MA, Boyko multiple sclerosis. BMC Neurol. (2013) 13:37. doi: 10.1186/1471-2377- A, et al. Multiple sclerosis registries in Europe - results of a systematic 13-37 survey. Mult Scler. (2014) 20:1523–32. doi: 10.1177/13524585145 20. Gershon RC, Lai JS, Bode R, Choi S, Moy C, Bleck T, et al. Neuro-QoL: quality 28760 of life item banks for adults with neurological disorders: item development 10. Institute of Medicine. Best Care at Lower Cost. The Path to Continuously and calibrations based upon clinical and general population testing. Qual Life Learning Health Care in America. Washington, DC: The National Academies Res. (2012) 21:475–86. doi: 10.1007/s11136-011-9958-8 Press. (2013). 21. Miller DM, Bethoux F, Victorson D, Nowinski CJ, Buono S, 11. Rhodes JK, Schindler D, Rao SM, Venegas F, Bruzik ET, Gabel W, et al. Lai JS, et al. Validating neuro-QoL short forms and targeted Multiple sclerosis performance test: technical development and usability. Adv scales with people who have multiple sclerosis. Mult Scler. (2016) Ther. (2019) 36:1741–55. doi: 10.1007/s12325-019-00958-x 22:830–41. doi: 10.1177/1352458515599450 Frontiers in Neurology | www.frontiersin.org 14 August 2020 | Volume 11 | Article 632 Mowry et al. Learning Health System in MS 22. Cella D, Lai JS, Nowinski CJ, Victorson D, Peterman A, Miller part of this program based on fair market value. EM reports research support D, et al. Neuro-QoL: brief measures of health-related quality from Biogen (MS PATHS and investigator-initiated studies) and Sanofi-Genzyme of life for clinical research in neurology. Neurology. (2012) for investigator-initiated studies, serving as site principal investigator for a Sun 78:1860–7. doi: 10.1212/WNL.0b013e318258f744 Pharma study, free medication from Teva Neuroscience for use in a clinical 23. Rudick RA, Cutter G, Reingold S. The multiple sclerosis functional composite: trial, for which she is principal investigator, and royalties for editorial duties a new clinical outcome measure for multiple sclerosis trials. Mult Scler. (2002) from UpToDate. RB reports consulting fees from Biogen, Genentech/Roche, 8:359–65. doi: 10.1191/1352458502ms845oa Novartis, and Sanofi-Genzyme, research support from Biogen, and personal and 24. Traboulsee A, Simon JH, Stone L, Fisher E, Jones DE, Malhotra A, et al. institutional equity ownership related to the MSPT licensed to Biogen and Qr8 Revised recommendations of the Consortium of MS Centers task force for by Cleveland Clinic. JW, CM, EF, BK, FP, and RR are employees of and hold a standardized MRI protocol and clinical guidelines for the diagnosis and stock/stock options in Biogen. TB reports research support from Avid/Lilly and follow-up of multiple sclerosis. AJNR Am J Neuroradiol. (2016) 37:394– Roche, travel support from Biogen, and serving as clinical trial investigator for 401. doi: 10.3174/ajnr.A4539 Avid/Lilly, Biogen, and Roche. CH reports speaking and consulting fees from 25. Calabresi PA, Kieseier BC, Arnold DL, Balcer LJ, Boyko A, Pelletier J, Biogen, EMD Serono, Genentech, Genzyme, and Novartis, research support et al. Pegylated interferon beta-1a for relapsing-remitting multiple sclerosis from Biogen, Genentech, and Patient-Centered Outcomes Research Institute, (ADVANCE): a randomised, phase 3, double-blind study. Lancet Neurol. and research support to her institution, Cleveland Clinic, from Biogen. MH (2014) 13:657–65. doi: 10.1016/S1474-4422(14)70068-7 reports research support from Biogen, Chugai, National Institutes of Health 26. Fox RJ, Miller DH, Phillips JT, Hutchinson M, Havrdova E, Kita M, et al. (NeuroNEXT), Novartis, and Patient-Centered Outcomes Research Institute. II Placebo-controlled phase 3 study of oral BG-12 or glatiramer in multiple reports funding from Biogen in the form of a contract with her employer, Johns sclerosis. N Engl J Med. (2012) 367:1087–97. doi: 10.1056/NEJMoa1206328 Hopkins University (JHU), to support effort on this project, serving as principal 27. Gold R, Kappos L, Arnold DL, Bar-Or A, Giovannoni G, Selmaj K, et al. investigator on a research grant from Siemens paid to JHU for an unrelated project, Placebo-controlled phase 3 study of oral BG-12 for relapsing multiple and consultant fees from Alexion. SJ reports speaker fees from St. Jude Hospital, sclerosis. N Engl J Med. (2012) 367:1098–107. doi: 10.1056/NEJMoa1114287 travel support from Siemens, grant support from Biogen for MS PATHS, and 28. Polman CH, O’Connor PW, Havrdova E, Hutchinson M, Kappos grant support from St. Jude Hospital. HK reports research support from Novartis L, Miller DH, et al. A randomized, placebo-controlled trial of and traveling and speaking/consulting fees and/or honoraria from Novartis, natalizumab for relapsing multiple sclerosis. N Engl J Med. (2006) Biogen, Teva, IXICO, and Siemens. LK reports consulting fees from Biogen, 354:899–910. doi: 10.1056/NEJMoa044397 Gerson Lehrman, Novartis, and Sanofi, grant support from Biogen, and receipt 29. Rudick RA, Stuart WH, Calabresi PA, Confavreux C, Galetta SL, Radue EW, of royalties for the Fatigue Severity Scale licensed to various biopharmaceutical et al. Natalizumab plus interferon beta-1a for relapsing multiple sclerosis. N entities. YL declares no competing interests. XM reports speaker fees/travel Engl J Med. (2006) 354:911–23. doi: 10.1056/NEJMoa044396 expenses for scientific meetings or steering committees/advisory boards for 30. Debray TP, Vergouwe Y, Koffijberg H, Nieboer D, Steyerberg EW, Moons clinical trials for Actelion, Biogen, Celgene, EXCEMED, Genzyme, Merck KGM. A new framework to enhance the interpretation of external validation Serono, MS International Foundation, National MS Society, Novartis, Sanofi- studies of clinical prediction models. J Clin Epidemiol. (2015) 68:279– Genzyme, Roche, and Teva. RN reports consulting fees and/or honoraria 89. doi: 10.1016/j.jclinepi.2014.06.018 from Alexion, Alkermes, Biogen, Celgene, EMD Serono, Genentech, Genzyme, 31. D’Agostino RB Jr. Propensity score methods for bias NervGen, Novartis, TG Therapeutics, and Third Rock Ventures. JN reports reduction in the comparison of a treatment to a non- consulting/speaking honoraria from Biogen, EMD Serono, Genentech, Genzyme, randomized control group. Stat Med. (1998) 17:2265– and Novartis. AR serves on scientific advisory boards for Bayer, Biogen, Icometrix, 81. doi: 10.1002/(SICI)1097-02581998101517:19<2265::AID-SIM918> Olea Medical, and SyntheticMR, is a consultant/speaker for Novartis, Roche, and 3.0.CO;2-B Sanofi-Genzyme, and receives research support from Biogen. MS reports research 32. Parsons LS. Reducing bias in a propensity score matched pair sample using funding for the conduct of MS PATHS from Biogen. BT reports advisory boards greedy matching techniques. In: Proceedings of the Twenty-Sixth Annual SAS for Bayer, Biogen, CSL Behring, Genzyme, Grifols, Novartis, and UCB, speaker Users Group International Conference. Cary, NC: SAS Institute (2004). honoraria from Bayer, Biogen, CSL Behring, Genzyme, Grifols, Novartis, and 33. European Medicines Agency. Report on Multiple Sclerosis Registries Octapharma, consulting fees from Bayer, Biogen, CSL Behring, Genzyme, Grifols, - Workshop 7 July 2017: Patient Registry Initiative. (2017) Available Novartis, and UCB, and research support from Biogen and Novartis. MTin online at: https://www.ema.europa.eu/en/documents/report/report-multiple- reports consulting/speaking honoraria from Almirall, Bayer Schering, Biogen, sclerosis-registries_en.pdf (accessed May 3, 2019). Genzyme, Merck Serono, Novartis, Roche, Sanofi, and Teva, grants from Biogen, 34. European Medicines Agency. Patient Registries Workshop, 28 October Genzyme, and Novartis, and serving as co-editor of Multiple Sclerosis Journal 2016: Observations and Recommendations Arising From The Workshop. – Experimental, Translational, and Clinical. MTiv reports funding from Biogen (2017) Available online at: https://www.ema.europa.eu/en/documents/report/ in the form of a contract with her employer, University of Rochester, to support report-patient-registries-workshop_en.pdf (accessed May 3, 2019). efforts on MS PATHS. TZ reports speaker fees/travel expenses for scientific 35. European Medicines Agency. EMA Regulatory Science To 2025: Strategic meetings or steering committees/advisory boards for clinical trials for Almirall, Reflection. (2018) Available online at: https://www.ema.europa.eu/en/ Bayer, Biogen, Celgene, Genzyme, Gilead, Merck Serono, Novartis, Roche, and documents/regulatory-procedural-guideline/ema-regulatory-science-2025- Teva and grant support from BAT, Biogen, Genzyme, Merck Serono, Novartis, strategic-reflection_en.pdf (accessed May 3, 2019). and Roche. 36. Avasarala J. Inadequacy of clinical trial designs and data to control for the confounding impact of race/ethnicity in response to treatment in multiple The handling editor declared a past co-authorship with AR. sclerosis. JAMA Neurol. (2014) 71:943–4. doi: 10.1001/jamaneurol.2014.79 37. Ziemssen T, Kern R, Thomas K. Multiple sclerosis: clinical profiling and data Copyright © 2020 Mowry, Bermel, Williams, Benzinger, de Moor, Fisher, Hersh, collection as prerequisite for personalized medicine approach. BMC Neurol. Hyland, Izbudak, Jones, Kieseier, Kitzler, Krupp, Lui, Montalban, Naismith, (2016) 16:124. doi: 10.1186/s12883-016-0639-7 Nicholas, Pellegrini, Rovira, Schulze, Tackenberg, Tintore, Tivarus, Ziemssen 38. Nazareth TA, Rava AR, Polyakov JL, Banfe EN, Waltrip II RW, Zerkowski and Rudick. This is an open-access article distributed under the terms of KB, et al. Relapse prevalence, symptoms, and health care engagement: patient the Creative Commons Attribution License (CC BY). The use, distribution insights from the multiple sclerosis in America 2017 survey. Mult Scler Relat or reproduction in other forums is permitted, provided the original author(s) Disord. (2018) 26:219–34. doi: 10.1016/j.msard.2018.09.002 and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, Conflict of Interest: The participating healthcare institution of each investigator distribution or reproduction is permitted which does not comply with these receives financial compensation for data shared and biosamples collected as terms. Frontiers in Neurology | www.frontiersin.org 15 August 2020 | Volume 11 | Article 632