ORIGINAL RESEARCH published: 09 August 2021 doi: 10.3389/fneur.2021.701946 Neuropsychological Assessments of Patients With Acquired Brain Injury: A Cluster Analysis Approach to Address Heterogeneity in Web-Based Cognitive Rehabilitation Alejandro García-Rudolph 1,2,3*, Alberto García-Molina 1,2,3 , Eloy Opisso 1,2,3 , Josep María Tormos 1,2,3 , Vince I. Madai 4,5,6 , Dietmar Frey 4 and Montserrat Bernabeu 1,2,3 1 Universitat Autònoma de Barcelona, Cerdanyola del Vallès, Spain, 2 Fundació Institute d’Investigació en Ciències de la Salut Germans Trias i Pujol, Badalona, Spain, 3 Institut Guttmann Hospital de Neurorehabilitacio, Badalona, Spain, 4 CLAIM Charité Lab for AI in Medicine, Charité Universitätsmedizin Berlin, Berlin, Germany, 5 QUEST Center for Transforming Biomedical Research, Berlin Institute of Health (BIH), Berlin, Germany, 6 Faculty of Computing, Engineering and the Built Environment, Edited by: School of Computing and Digital Technology, Birmingham City University, Birmingham, United Kingdom Guoqiang Xing, Affiliated Hospital of North Sichuan Medical College, China We aimed to (1) apply cluster analysis techniques to mixed-type data (numerical and Reviewed by: categorical) from baseline neuropsychological standard and widely used assessments of Robert Perna, patients with acquired brain injury (ABI) (2) apply state-of-the-art cluster validity indexes University of Michigan Medical Center, United States (CVI) to assess their internal validity (3) study their external validity considering relevant Nabil Kitchener, aspects of ABI rehabilitation such as functional independence measure (FIM) in activities General Organization for Teaching of daily life assessment (4) characterize the identified profiles by using demographic Hospitals and Institutes, Egypt and clinically relevant variables and (5) extend the external validation of the obtained *Correspondence: Alejandro García-Rudolph clusters to all cognitive rehabilitation tasks executed by the participants in a web-based alejandropablogarcia@gmail.com cognitive rehabilitation platform (GNPT). We analyzed 1,107 patients with ABI, 58.1% traumatic brain injury (TBI), 21.8% stroke and 20.1% other ABIs (e.g., brain tumors, Specialty section: This article was submitted to anoxia, infections) that have undergone inpatient GNPT cognitive rehabilitation from Neurorehabilitation, September 2008 to January 2021. We applied the k-prototypes algorithm from the a section of the journal clustMixType R package. We optimized seven CVIs and applied bootstrap resampling Frontiers in Neurology to assess clusters stability (fpc R package). Clusters’ post hoc comparisons were Received: 29 April 2021 Accepted: 02 July 2021 performed using the Wilcoxon ranked test, paired t-test or Chi-square test when Published: 09 August 2021 appropriate. We identified a three-clusters optimal solution, with strong stability (>0.85) Citation: and structure (e.g., Silhouette > 0.60, Gamma > 0.83), characterized by distinctive level García-Rudolph A, García-Molina A, Opisso E, Tormos JM, Madai VI, of performance in all neuropsychological tests, demographics, FIM, response to GNPT Frey D and Bernabeu M (2021) tasks and tests normative data (e.g., the 3 min cut-off in Trail Making Test-B). Cluster 1 Neuropsychological Assessments of was characterized by severe cognitive impairment (N = 254, 22.9%) the mean age was Patients With Acquired Brain Injury: A Cluster Analysis Approach to Address 47 years, 68.5% patients with TBI and 22% with stroke. Cluster 2 was characterized by Heterogeneity in Web-Based mild cognitive impairment (N = 376, 33.9%) mean age 54 years, 53.5% patients with Cognitive Rehabilitation. Front. Neurol. 12:701946. stroke and 27% other ABI. Cluster 3, moderate cognitive impairment (N = 477, 43.2%) doi: 10.3389/fneur.2021.701946 mean age 33 years, 83% patients with TBI and 14% other ABI. Post hoc analysis on Frontiers in Neurology | www.frontiersin.org 1 August 2021 | Volume 12 | Article 701946 García-Rudolph et al. Clustering Neuropsychological Assessments cognitive FIM supported a significant higher performance of Cluster 2 vs. Cluster 3 (p < 0.001), Cluster 2 vs. Cluster 1 (p < 0.001) and Cluster 3 vs. Cluster 1 (p < 0.001). All patients executed 286,798 GNPT tasks, with performance significantly higher in Cluster 2 and 3 vs. Cluster 1 (p < 0.001). Keywords: acquired brain injury, cluster analysis, cognitive rehabilitation, web-based platform, neuropsychological assessment, mixed-type data, functional independence INTRODUCTION traditionally implemented using commercial software packages as opposed to open access libraries (14, 15). Cognitive rehabilitation is widely recognized as a standard Therefore, in this work we aimed to (1) apply CA techniques element of rehabilitation services for patients with acquired brain to mixed data types (categorical and numerical variables) from injury (ABI) in several international clinical guidelines [e.g., baseline neuropsychological assessments of patients with ABI the European Federation of Neurological Sciences (1) or the (2) apply seven different state-of-the-art CVIs for assessing Brain Injury Special Interest Group of the American Congress of clusters’ internal validity to identify the optimal number of Rehabilitation Medicine (2)]. clusters considering different criteria (3) once an optimal New strategies for providing cognitive rehabilitation number of clusters is selected, study their external validity, programs are required and continuously being integrated considering (i) the baseline assessments not used for creating into clinical practice (3). One of such strategies is the the clusters, (ii) other clinical and demographic variables incorporation of web-based systems, with several of them relevant to ABI rehabilitation (such as gender or time since having already contributed to optimizing cognitive interventions injury to assessment) and (iii) motor and cognitive functional (4, 5). Due to the relative recent arrival of such services, independence measure (FIM) in activities of daily living (4) best strategies for integrating them into everyday clinical characterize the identified profiles by using demographic and practice are still in progress (6). Nevertheless, those aiming clinically relevant variables and (5) extend the external validation for personalization of the proposed rehabilitation activities of the obtained clusters to all cognitive rehabilitation tasks to patients according to their specific needs, appear to be executed by the participants included in the study along more effective (7). It has been previously reported that their whole rehabilitation process in a web-based cognitive approaches that: (1) use a baseline cognitive evaluation rehabilitation platform. founded on standardized neuropsychological tests to Guttmann, NeuroPersonalTrainerr (GNPT) (16) is the personalize the therapeutic interventions (2) offer immediate web-based cognitive rehabilitation platform used in the task-specific feedback to end-users and (3) dynamically present study. At admission to cognitive rehabilitation using adjust the rehabilitation scheme accordingly, are most GNPT, every patient is assessed using a comprehensive effective (8). standardized battery of neuropsychological tests (baseline The heterogeneity of ABI is extensively considered as one assessment). The actual GNPT implementation integrates of the most significant barriers to finding effective therapeutic an automatic therapy planning functionality, the Intelligent interventions (9). Identifying subgroups of patients who have Therapy Assistant (ITA) (17). The ITA provides therapists with distinguishable cognitive profiles that, in turn can assist a recommended schedule of cognitive tasks to be executed in treatment planning and patient care, is crucial. Cluster by each patient during a given period of time. Therapists analysis (CA) allows for the identification of homogeneous can in turn manually modify the proposed schedule of tasks subgroups where cognitive heterogeneity is present, based to their own preferences, according to their criteria (for on similarities in performance on baseline neuropsychological example changing their order or modifying their execution tests (10, 11). parameters). In order to propose such schedule of tasks, the The application of CA to baseline assessments of patients ITA takes as starting point a set of patient’s cognitive profiles, with ABI, still presents several limitations, despite the extensive obtained using CA from the baseline neuropsychological production of research literature for over 25 years: (i) lack of assessment (17). internal validation using standardized cluster validity indices Therefore, the main clinical implication of this work is (CVI) (12) (ii) CA has been applied as isolated instances, barely to provide a validated (internally and externally) hands-on integrated in the context of web-based cognitive rehabilitation approach based on standardized neuropsychological baseline (11) (iii) lack of external validation of the obtained clusters assessments and CA methods, implemented using open access (e.g., considering patients’ performance in activities of daily software libraries. The proposed approach can thus be applied living) (iv) reduced samples (rarely larger than n = 500) (v) in the context of other web-based cognitive platforms that CA has been applied to baseline assessments of individual usually integrate an initial neuropsychological profiling of the cognitive functions, therefore not allowing for a comprehensive patients to overcome heterogeneity. Extensions to similar web- description of patient’s profiles (13) (vi) CA has been mostly based platforms (18) and to other populations with cognitive applied to numerical variables (vii) CA methods have been impairments may also build on this work. Frontiers in Neurology | www.frontiersin.org 2 August 2021 | Volume 12 | Article 701946 García-Rudolph et al. Clustering Neuropsychological Assessments MATERIALS AND METHODS neuropsychological assessment and 231 with more than 365 years since injury to neuropsychological assessment before starting Subject Selection GNPT. Finally, from the 1,171 resulting patients, 64 had more All patients were consecutively admitted to the rehabilitation unit than one GNPT identifier for the same patient, therefore we kept of the ABI Department of Institut Guttmann, Barcelona Spain, only the first one of them, leaving the 1,107 patients included in between January 2008 and December 2020. Institut Guttmann is this study (Figure 1). a specialized clinical center certified in quality of care and patient safety (Joint Commission International since 2005, consecutively recertified in 2009, 2012 and 2018) (19). Procedure and Neuropsychological From an initial cohort of 2,312 patients with ABI, we enrolled Measures at Admission only those who fulfilled the following inclusion criteria: (1) This was a retrospective observational study conforming to Traumatic brain injury (TBI), ischemic or hemorrhagic stroke or the STROBE Guidelines (“Strengthening the Reporting of other ABI (brain tumor, anoxia,. . . ) identified based on medical Observational Studies in Epidemiology”) (20). records relative to the acute phase of the intensive care unit At admission to the rehabilitation unit of the ABI Department (ICU) period; (2) age ≥18 years; and (3) first admission to of Institut Guttmann, each patient is assigned a medical neurorehabilitation unit. Exclusion criteria were: (1) the presence doctor, who coordinates the rehabilitation team (a nurse, of aphasia; (2) the presence of a premorbid history of psychiatric a neuropsychologist, a physiotherapist, an occupational disease or other severe disability (e.g., tetraplegia, paraplegia; therapist, a social worker and a clinical psychologist based and (3) time since injury to GNPT rehabilitation admission on the characteristics of the case). Therefore, admission >365 days. neuropsychological assessments (as well as all clinical and From the initial group, 684 patients with aphasia and 14 demographic data analyzed in this study) are systematically with other severe disability were removed. In the resulting 1,614 recorded in the electronic health records of the hospital. Such patients 212 were younger than 18 years old at the moment of standard neuropsychological assessments (summarized in FIGURE 1 | Participants’ selection flowchart. Frontiers in Neurology | www.frontiersin.org 3 August 2021 | Volume 12 | Article 701946 García-Rudolph et al. Clustering Neuropsychological Assessments TABLE 1 | Neuropsychological assessments at admission with the identifiers The participants were anonymized and non-identifiable. used for each of them along this study. A specific written informed consent was not required for Test Identifier Function/ participants to be included in this study, nevertheless at Subfunction admission participants provide written informed consent to be included in research studies addressed by the Institut Test Barcelona—Personal Orientation T1 Orientation/Personal Guttmann hospital. Test Barcelona—Spatial Orientation T2 Orientation/Spatial Test Barcelona—Temporal Orientation T3 Orientation/Temporal Web-Based Cognitive Rehabilitation WAIS-III—Direct digits T4 Attention/Selective The GNPT (34) web-based cognitive rehabilitation platform TMT–A T5 Attention/Selective used in this study is composed by a set of 149 different web- STROOP—words T6 Attention/Selective based cognitive rehabilitation tasks. There is not an established STROOP—color T7 Attention/Selective previous order in which patients should execute such tasks. STROOP—words/colors T8 Attention/Selective Therefore every patient executes (eventually) a different subset WAIS III—Keys T9 Attention/Selective of them in a different order during their rehabilitation process, WAIS III—Cubes T14 Executive functions/Planning taking between 2 and 6 months, distributed in 2 to 5 sessions WAIS III—Inverse digits T15 Memory/Working memory a week. During each session the patient executes between 4 WAIS III—Letters and numbers T16 Executive to 10 cognitive rehabilitation tasks, the total duration of one functions/Sequencing session ranges between 45 minutes and 1 hour. Each task mainly RAVLT immediate recall T17 Memory/verbal addresses one of the following functions: memory, executive RAVLT deferred recall T18 Memory/verbal functioning, attention, gnosias, calculus, orientation, language RAVLT recognition T19 Memory/verbal and social cognition. Immediately after each execution of a task, TMT–B T20 Executive functions/Flexibility the patient gets a feedback on performance (ranging from 0 to WCST—Categories T21 Executive 100, as the percentage of compliance), 0% being the lowest level functions/Categorization of compliance and 100% the highest (15). WCST—Perseverative errors T22 Executive functions/Flexibility STROOP—Interference T23 Executive functions/Inhibition Statistical Analysis 3rd WAIS III, Wechsler Adult Intelligence Scale version; TMT, Trail Making Test; RAVLT, Rey Statistical analyses were performed using R (version 3.5.31) (35) Auditory Verbal Learning Test; WCST, Wisconsin Card Sorting Test. a value of P < 0.05 is considered statistically significant. Data Pre-processing Table 1), were performed by trained neuropsychologists with Before running the CA methods, a pre-processing phase experience in neurological cognitive disorders, who were blind was performed involving all neuropsychological assessments to any other result. presented in Table 1. We run Spearman correlation analysis, Orientation was assessed using 3 specific items of the using the corrplot R package (36). We analyzed all pairs of Barcelona Test (21) i.e., personal (denoted throughout this work correlations and removed highly correlated variables (r > 0.5 as T1), spatial (T2) and temporal (T3) (22). P < 0.05) keeping at least one variable of each cognitive Attention was assessed using the Direct Digits (T4) of the function represented in Table 1 (orientation, attention, memory Wechsler Adult Intelligence Test–III (WAIS-III) (23), the Trail and executive functions) no pair of variables with a significant Making Test–part A (TMT A) (T5) (24), the Stroop test (25) correlation coefficient larger than 0.5 was kept. Words (T6), Colors (T7) and Words/colors (T8) (26) and the Pre-processing phase also included Z-normalization (37) of all Keys (T9) subtest of the WAIS III (27). the included numerical variables, using the scale () function of the Working memory was assessed using the WAIS III–Inverse base R package (38). digits (T15) and verbal memory using the Rey Auditory Verbal Memory Test (RAVLT) (28), specifically RAVLT learning (T17), Cluster Analysis Method: k-Prototypes RAVLT free-recall memory (T18) and RAVLT recognition For cluster analysis based on mixed-type data (i.e., data consisting (T19) (29). of numerical and categorical variables), comparatively few In relation to executive functions, planning was assessed using clustering methods are available. One popular approach to deal the WAIS III—Cubes (T14), sequencing was assessed with the with this kind of problems is an extension of the k-means WAIS III—Letters and numbers (T16), the Trail Making Test— algorithm (39), the so-called k-prototypes algorithm, in this work part B (TMT B) for flexibility (T20) (30). Categorization was we applied the implementation provided in the clustMixType R assessed using the Wisconsin Card Sorting Test (31) (WCST)— package (40). Categories (T21) and WCST—Perseverative errors (T22) (32) The algorithm iterates in a manner similar to the k-means and inhibition using the STROOP—Interference (T23) (33). algorithm where for the numeric variables the mean and for The authors confirm that this study is compliant with the the categorical variables the mode, minimizes the total within Helsinki Declaration of 1975, as revised in 2008 and it was cluster distance. approved by the Ethics Committee of Clinical Research of The steps of the implementation are based on the Huang’s Institut Guttmann. k-prototypes algorithm (41): Frontiers in Neurology | www.frontiersin.org 4 August 2021 | Volume 12 | Article 701946 García-Rudolph et al. Clustering Neuropsychological Assessments 1. Initialization with random cluster prototypes. TABLE 2 | Demographics and clinical characteristics at admission. 2. For each observation do: Variable N = 1,107 a. Assign observations to its closest prototype according to a Sex, male, n (%) 793 (71.6%) distance metric. Age 43.5 (14.8) b. Update cluster prototypes by cluster-specific means/modes Injury, n (%) for all variables. TBI 643 (58.1%) 3. As long as any observations have swapped their cluster Stroke 241 (21.8%) assignment in 2 or the maximum number of iterations has not Hemorrhagic 121 (50.2%) been reached: repeat from 2. Ischemic 120 (49.8%) Other ABI 223 (20.1%) Optimal Number of Clusters: Internal Validation Tumor 73 (32.7%) Anoxia 65 (29.1%) It is further known that the selection of a suitable number of Infection 21 (9.4%) clusters k is particularly crucial in partitioning cluster procedures. Iatrogenesis 15 (6.7%) Many implementations of cluster validation indices are not Other 49 (22.1%) suitable for mixed-type data as recently reported (42). TBI severity (GCS) 6.5 (3.4) We compared the internal validity of the k-prototypes Stroke severity (NIHSS) 10.8 (5.2) algorithm for different number of clusters (k = 2..6) using Test Barcelona—Personal Orientation (T1) 6.5 (1.5) 7 state-of-the-art cluster validation indices implemented in Test Barcelona—Spatial Orientation (T2) 4.3 (1.2) the clustMixType R package: Tau, Gamma, GPlus, McClain, Test Barcelona—Temporal Orientation (T3) 20.3 (5.8) PtBiseral, Silhouette, Sum of Squares (43). WAIS-III—Direct digits (T4) 5.2 (1.6) TMT–A (T5) 100.9 (84.4) Clusters Stability STROOP—words (T6) 70.1 (20.4) To assess whether a cluster represents true structure is to see STROOP—color (T7) 48.9 (14.8) if the cluster holds up under plausible variations in the dataset. STROOP—words/colors (T8) 28.6 (12.4) We used bootstrap resampling (44) from the fpc package (45) to WAIS III—Keys (T9) 37.8 (17.3) generate such perturbations in the input data and evaluate how WAIS III—Cubes (T14) 24.2 (12.0) stable the obtained clusters were. WAIS III—Inverse digits (T15) 3.7 (1.0) WAIS III—Letters and numbers (T16) 7.1 (2.9) External Validation RAVLT immediate recall (T17) 31.6 (11.3) RAVLT deferred recall (T18) 4.4 (3.8) In order to validate any cluster solution, it is important to RAVLT recognition (T19) 7.7 (5.1) compare the resulting clusters on variables that were not included TMT–B (T20) 225.2 (163.3) in the original clustering process (46). WCST—Categories (T21) 2.7 (2.4) We included external variables found in previous related WCST—Perseverative errors (T22) 41.4 (35.9) research (15) such as gender, age, age intervals, educational STROOP—Interference (T23) −0.4 (6.1) level or time since injury. Furthermore, ABI is a major cause Time since injury in days 93.5 (68.2) of long-term ADL disability (47) therefore we included a Time since injury in days (ranges), n (%) standard measure of functional independence extensively used 0–45 270 (24.4%) in population after ABI, the FIM (Functional Independence 46–90 410 (37.0%) Measure). The FIM is typically reported as a total score or 91–180 303 (27.4%) subdivided into a motor and a cognitive sub-score (48). External 181–364 124 (11.2%) validation also included all the GNPT tasks executed by all Education, n (%) participating patients during the period under study. Primary 518 (46.8%) Secondary 365 (33.0%) Post hoc Analyses University 224 (20.2%) We conducted post hoc comparisons considering both the TBI, Traumatic brain injury; ABI, Acquired brain injury; GCS, Glasgow Coma Scale; NIHSS, variables used and not used for building the clusters. We used National institute of Health Stroke Scale; WAIS III, Wechsler Adult Intelligence Scale the Wilcoxon ranked test, paired t-test or Chi-square test when 3rd version; TMT, Trail Making Test; RAVLT, Rey Auditory Verbal Learning Test; WCST, Wisconsin Card Sorting Test. appropriate. The Shapiro Wilk test was used to assess normality and the Levene test for homogeneity of variances. RESULTS reported, 58.1% were patients with TBI, 21.8% patients with stroke and 20.1% other acquired brain injuries. Demographic and Clinical Characteristics Severity of injury was reported for patients with TBI, mean at Baseline reported GCS was 6.5 (3.4) [regarded as severe (49)] and for In Table 2, demographic and neuropsychological variables patients with stroke mean NIHSS was 10.8 (5.2) [moderately collected in all participating patients at study entry are severe (50)]. Frontiers in Neurology | www.frontiersin.org 5 August 2021 | Volume 12 | Article 701946 García-Rudolph et al. Clustering Neuropsychological Assessments Preprocessing Phase: Correlation Analysis of injury (TBI, STROKE, OTHER ABI). After z-normalization of and z Normalization the numerical variables, these were used as input to the clustering Spearmans’ correlations for all pairs of variables were k-prototypes algorithm. performed as initial pre-processing phase. As detailed in Supplementary Table 1, a representative example is the Cluster Analysis: Internal Validation Barcelona Test, orientation variables were found to be highly Table 3 summarizes the results obtained for all ICVs with the k correlated: T1 and T2 (r = 0.56, p < 0.001); T2 and T3 (r = 0.63, number of clusters ranging from 2 to 6. A positive value (+) in p < 0.001), therefore only T3 was kept in the final set of variables, the Criteria column indicates that the higher the maximum value as shown in Supplementary Table 1 T3 is not correlated to of the index is used to indicate the optimal number of clusters. any other variable. Similarly, T5 is highly correlated with T6 (r Similarly, a negative value (–) in the Criteria column indicates = −0.60, p < 0.001), T7 (r = −0.57, p < 0.001), T9 (r = −0.72, that the minimum value of the index is used to indicate the p < 0.001), T20 (r = 0.79, p < 0.001) therefore only T5 was kept. optimal number of clusters. The final set of variables was: T3, T5, T14, T15, T18, T22, The solutions with k = 2 and 3 clusters clearly show optimal age at the moment of neuropsychological assessment and type values for all ICVs (as highlighted in bold in Table 3), but k = 3 TABLE 3 | ICV indexes obtained for the different number of clusters (k = 2..6). Validation index 2 3 4 5 6 Criteria Tau 0.5103 0.5823 0.4648 0.4908 0.4635 + Gamma 0.9844 0.8338 0.7060 0.7181 0.7474 + GPlus 0.0020 0.0405 0.0656 0.0472 0.0486 − McClain 0.0755 0.1068 0.1279 0.1177 0.1205 − PtBiseral 0.7826 0.3209 0.2954 0.2578 0.2262 + Silhouette 0.8926 0.6063 0.4199 0.4359 0.3758 + Sum of Squares 731,194 553,509 450,482 394,372 356,340 − TABLE 4 | Kproto results considering only the variables used for creating the clusters with post hoc analysis. Variable Cluster 1 (N = 254) Cluster 2 (N = 376) Cluster 3 (N = 477) (N = 670) Post hoc comparisons p Age 47.6 (14.7) 54.1 (9.4) 32.9 (10.7) Cluster 1 vs. Cluster 2 < 0.001 Cluster 1 vs. Cluster 3 < 0.001 Cluster 2 vs. Cluster 3 < 0.001 T3 12.1 (9.1) 22.3 (1.7) 22.0 (2.1) Cluster 1 vs. Cluster 2 < 0.001 Cluster 1 vs. Cluster 3 < 0.001 Cluster 2 vs. Cluster 3 0.0048 T5 250.1 (80.8) 68.1 (42.9) 75.4 (48.6) Cluster 1 vs. Cluster 2 < 0.001 Cluster 1 vs. Cluster 3 < 0.001 Cluster 2 vs. Cluster 3 0.01777 T14 13.3 (6.9) 25.9 (11.6) 24.9 (12.1) Cluster 1 vs. Cluster 2 < 0.001 Cluster 1 vs. Cluster 3 < 0.001 Cluster 2 vs. Cluster 3 0.1965 T15 2.9 (0.7) 4.1 (0.9) 3.6 (0.9) Cluster 1 vs. Cluster 2 < 0.001 Cluster 1 vs. Cluster 3 < 0.001 Cluster 2 vs. Cluster 3 < 0.001 T18 1.0 (1.8) 6.8 (3.6) 3.7 (3.2) Cluster 1 vs. Cluster 2 < 0.001 Cluster 1 vs. Cluster 3 < 0.001 Cluster 2 vs. Cluster 3 < 0.001 T22 95.9 (13.6) 20.1 (14.8) 37.6 (32.2) Cluster 1 vs. Cluster 2 < 0.001 Cluster 1 vs. Cluster 3 < 0.001 Cluster 2 vs. Cluster 3 < 0.001 Injury, n (%) TBI 174 (68.5%) 73 (19.4%) 396 (83.0%) < 0.001 STROKE 24 (9.4%) 201 (53.5%) 16 (3.4%) OTHER ABI 56 (22.0%) 102 (27.1%) 65 (13.6%) Frontiers in Neurology | www.frontiersin.org 6 August 2021 | Volume 12 | Article 701946 García-Rudolph et al. Clustering Neuropsychological Assessments TABLE 5 | Kproto results considering the variables not used for creating the TABLE 6 | Post hoc comparisons of Cluster 2 and Cluster 3 of the variables not clusters. used for creating the clusters. Variables Cluster 1 Cluster 2 Cluster 3 Variables Cluster 2 Cluster 3 p (N = 254) (N = 376) (N = 477) (N = 376) (N = 477) (N = 670) (N = 670) Sex, male, n (%) 185 (72.8%) 251 (66.8%) 357 (74.8%) T1 6.9 (0.5) 6.8 (0.4) 0.02073 Age range, n (%) T2 4.7 (0.6) 4.6 (0.7) 0.002147 17–30 46 (18.1%) 7 (1.9%) 228 (47.8%) T4 5.8 (1.2) 5.4 (1.1) < 0.001 31–55 129 (50.8%) 202 (53.7%) 235 (49.3%) T6 75.0 (17.9) 69.4 (19.8) 0.0112 ≥56 79 (31.1%) 167 (44.4%) 14 (2.9%) T7 52.1 (13.3) 48.8 (14.4) 0.02537 T1 5.0 (2.9) 6.9 (0.5) 6.8 (0.4) T8 30.4 (11.0) 28.6 (12.9) 0.1004 T2 2.9 (2.0) 4.7 (0.6) 4.6 (0.7) T9 41.6 (17.7) 37.5 (15.8) 0.0117 T4 3.6 (2.3) 5.8 (1.2) 5.4 (1.1) T13 18.8 (2.4) 18.8 (2.3) 0.8172 T6 53.3 (24.1) 75.0 (17.9) 69.4 (19.8) T16 8.0 (2.5) 6.7 (2.8) < 0.001 T7 36.7 (16.6) 52.1 (13.3) 48.8 (14.4) T17 37.1 (10.2) 29.5 (10.3) < 0.001 T8 21.3 (12.9) 30.4 (11.0) 28.6 (12.9) T19 10.4 (4.2) 7.3 (4.8) < 0.001 T9 22.4 (16.1) 41.6 (17.7) 37.5 (15.8) T20 142.5 (81.8) 209.6 (147.9) < 0.001 T13 16.7 (3.8) 18.8 (2.4) 18.8 (2.3) T21 3.7 (2.2) 2.7 (2.4) < 0.001 T16 5.0 (2.9) 8.0 (2.5) 6.7 (2.8) T23 1.6 (2.3) 1.2 (4.9) < 0.001 T17 22.0 (8.7) 37.1 (10.2) 29.5 (10.3) T25 84.8 (79.8) 85.1 (79.1) 0.367 T19 2.7 (3.9) 10.4 (4.2) 7.3 (4.8) TSO 90.7 (73.7) 91.7 (63.9) 0.07775 T20 480.8 (95.8) 142.5 (81.8) 209.6 (147.9) TSO, Time since onset to rehab assessment. T21 0.2 (0.8) 3.7 (2.2) 2.7 (2.4) T23 1.5 (3.2) 1.6 (2.3) 1.2 (4.9) T25 162.3 (140.7) 84.8 (79.8) 85.1 (79.1) TSO 101.2 (67.1) 90.7 (73.7) 91.7 (63.9) Cluster Analysis: External Validation TSO range n (%) We performed twofold external validation: (1) by using 0–45 39 (15.4%) 120 (31.9%) 111 (23.3%) demographic (age, gender, education level, age ranges), clinical 46–90 97 (38.2%) 125 (33.2%) 188 (39.4%) variables (T1, T2, T4, T6, T7, T8, T9, T13, T16, T17, T19, T20, 91–180 91 (35.8%) 87 (23.1%) 125 (26.2%) T21, T23, T25) and then by using the total FIM, cognitive FIM 181–364 27 (10.6%) 44 (11.7%) 53 (11.1%) and motor FIM subtotals at admission and (2) considering all studies cognitive tasks executed by the patients in GNPT during the Primary (6 years) 139 (54.7%) 162 (43.1%) 217 (45.5%) period under study. Table 5 presents Kproto results considering Secondary (11 years) 66 (26.0%) 119 (31.6%) 180 (37.7%) the variables presented in Table 2 but not used for creating Tertiary (12 + years) 49 (19.3%) 95 (25.3%) 80 (16.8%) the clusters. As presented in Table 5, mean TMT-B (T20) values were 480.8 TSO, Time since onset to rehab assessment. (95.8) for Cluster 1, 142.5 (81.8) for Cluster 2 and 209.6 (147.9) for Cluster 3. Therefore, Cluster 1 is remarkably lower but both Cluster 2 and 3 higher than the normative value for Spanish outperforms k = 2 in the sum of squares index. Therefore a 3 people with TBI. It is important to remark that the percentage clusters solution is proposed in this work. of patients with TBI in each of the obtained clusters was 68.5% in Supplementary Figures 1–7 show the graphical Cluster 1, 19.4% in Cluster 2 and 83.0% in Cluster 3. representation of each of the ICV indexes for k = 2..6. Table 6 presents the post hoc comparisons of Cluster 2 vs. Solutions with higher k values were also tested, but did not Cluster 3 of the variables not used for creating the clusters. improve the obtained performances. Table 7 presents the external validation using the total FIM, cognitive FIM and motor FIM for N = 947 patients, Cluster Analysis: k-Prototypes Results therefore FIM was available for 85% of the initial 1,107 Table 4 presents the k-prototypes results considering only the patients. Post hoc analysis shows significant differences between variables used for creating the clusters and the corresponding all three clusters for cognitive FIM (details are presented in post hoc comparisons. Supplementary Figure 8). When comparing motor and total FIM, significant differences were found between Cluster 1 vs Stability Cluster 2, Cluster 1 vs Cluster 3, but not between Cluster 2 vs We analyzed perturbations on input data considering B = 100 Cluster 3. resampling runs, obtaining stable results. Vector of obtained Cluster 2 and Cluster 3 were significantly different in cognitive cluster stabilities for Cluster1, Cluster2 and Cluster3 = [.0.88, FIM, median and IQR values were 29.0 (22.0−32.0) and 26.0 0.87, 0.90]. (20.0–31.0) respectively, with mean values 26.6 (7.6) and 24.4 Frontiers in Neurology | www.frontiersin.org 7 August 2021 | Volume 12 | Article 701946 García-Rudolph et al. Clustering Neuropsychological Assessments TABLE 7 | FIM post hoc analysis. Variable Cluster 1 (N = 213) Cluster 2 (N = 329) Cluster 3 (N = 405) (N = 670) Post hoc comparisons p Age 50.0 (13.7) 54.4 (9.3) 33.0 (10.6) Cluster 1 vs. Cluster 2 < 0.001 Cluster 1 vs. Cluster 3 < 0.001 Cluster 2 vs. Cluster 3 < 0.001 TOTAL FIM 69.9 (33.6) 79.7 (29.7) 79.9 (33.3) Cluster 1 vs. Cluster 2 < 0.001 Cluster 1 vs. Cluster 3 < 0.001 Cluster 2 vs. Cluster 3 0.621 COGNITIVE FIM 21.7 (8.5) 26.6 (7.6) 24.4 (8.2) Cluster 1 vs. Cluster 2 < 0.001 Cluster 1 vs. Cluster 3 < 0.001 Cluster 2 vs. Cluster 3 < 0.001 MOTOR FIM 48.2 (27.2) 53.1 (24.7) 55.5 (27.0) Cluster 1 vs. Cluster 2 0.035 Cluster 1 vs. Cluster 3 0.003 Cluster 2 vs. Cluster 3 0.156 FIM, Functional Independence Measure. TABLE 8 | GNPT tasks executions by cluster. DISCUSSION Variable Cluster 1 Cluster 2 Cluster 3 In this work we extended the application of CA techniques (N = 66,933) (N = 81,031) (N = 138,834) to mixed data variables from baseline neuropsychological (N = 670) assessments of patients with ABI. We applied several state- of-the-art CVI involving such new set of categorical variables result 65.1 (33.8) 65.5 (35.4) 68.7 (33.7) to assess clusters’ internal validity. We studied the external 76.4 (41.1–95.0) 80.0 (40.0–97.0) 82.0 (50.0–97.2) validity of the obtained clusters considering relevant aspects of Memory 26,177 (39.1%) 36,117 (44.6%) 54,944 (39.6%) ABI rehabilitation such functional independence in activities Executive 15,359 (22.9%) 23,101 (28.5%) 38,350 (27.6%) of daily life and characterized the identified profiles by using ATTENTION 15,742 (23.5%) 14,421 (17.8%) 29,417 (21.2%) demographic and clinically relevant variables. Finally, clusters’ Language 5,333 (8.0%) 2,494 (3.1%) 6,597 (4.8%) characterization was confirmed using all cognitive rehabilitation Calculus 2,288 (3.4%) 2,048 (2.5%) 3,184 (2.3%) tasks executed by the patients included in the study along their Gnosias 1,047 (1.6%) 1,614 (2.0%) 3,293 (2.4%) whole rehabilitation process in a web-based GNPT cognitive Orientation 628 (0.9%) 782 (1.0%) 1,796 (1.3%) rehabilitation platform. Socialization 359 (0.5%) 454 (0.6%) 1,253 (0.9%) CA has been previously applied to the neuropsychological assessments presented in Table 1, the number of identified clusters varied from 3 to 6, to our best knowledge most of them (8.2) (p < 0.001) as presented in Table 7. Cognitive FIM cut- addressing patients with TBI, very few of them included patients off value for home discharge was previously reported as 23.5 with stroke or with other ABI. Besides, very few of them involving points, with a sensitivity of 73.7% and a specificity of 80.6% a sample larger than 300, none of them presented the CA as a (51). Therefore, the mean reported value for patients in Cluster component of a web-based cognitive rehabilitation service, very 3 was less than one point above the cut-off for home discharge, few of them considered more than one CVIs, none of them meanwhile participants in Cluster 2 were more than three points performed external validation using activities of daily living, above it. neither cognitive rehabilitation tasks. As presented in Table 6 significant differences were found For example, Thaler et al. (52) used CA to address between Cluster 2 and Cluster 3 in post hoc comparisons of TMT- heterogeneity in TMT A and B in participants with mild TBI B (T20). Differences with Cluster 1 were not reported in Table 6 (N=78). Three clusters were identified, two of them characterized because they were significantly lower in all tests, also as confirmed by both TMT scores and the third with a single performance in the cognitive, motor and total FIM in Table 7. pattern characterized by lower scores on TMT B. Clusters did not Table 8 presents Kproto results considering the total number differ on demographic or other clinical variables. of GNPT tasks executed by the 1,107 participants. The total Harman-Smith et al. (53) analyzed all subtests of the WAIS- number of executed GNPT tasks were 286,798 with 66,933 III from patients with mild, moderate, and severe TBI (n=220). executed by patients from Cluster 1, 81,031 executed by patients They produced two solutions, one with 4 and the other with from Cluster 2 and 138,834 executed by patients from Cluster 3. 6 clusters. According to the authors the latter better captures Figure 2 plots the mean weekly results obtained by all subtle variations in cognitive functioning. The 6 clusters differed participants grouped in Cluster 1 (red) and Cluster 3 (blue), in the levels and profiles of cognitive performance, self-reported showing a consistent higher performance by participants from recovery, education and injury severity. Cluster 3 during the whole period under study. Frontiers in Neurology | www.frontiersin.org 8 August 2021 | Volume 12 | Article 701946 García-Rudolph et al. Clustering Neuropsychological Assessments FIGURE 2 | Mean weekly results obtained by all participants grouped in Cluster 1 (red) and Cluster 3 (blue). Sherer et al. (54) applied CA to an Australian cohort of difference between them was the severity of impairment, both persons with TBI (N = 170) using the Wechsler Letter-Number at the single verb and at the sentence level, in production and Sequencing and Coding, the Rey Auditory Verbal Learning in comprehension. Test and the Trail Making Test. They identified a 5-clusters In relation to patients with other acquired brain injuries, to solution that largely replicated another cohort of persons with the best of our knowledge, previous CA research has not been TBI previously presented by Sherer et al. (55). applied using any of the neuropsychological tests presented in Zimmermann et al. (56) analyzed a Brazilian cohort of Table 1. CA applications addressed for example brain tumors outpatients with mild and moderate/ severe TBI (N=84) using segmentation of MRI images (60, 61), monitoring glioma the TMT, the Modified Wisconsin Card Sorting Test−48 cards, heterogeneity during tumor growth (62) or region of interest Verbal fluency measured using the Montreal Communication analysis based on Mass Spectrometry Imaging (63). Assessment and Auditory oral word span in sentences. Three As shown in Tables 4–8, in our case CA yielded to 3 clusters emerged and were characterized by deficits in: (1) clearly different clusters characterized by level of performance in inhibition, flexibility and focused attention; (2) inhibition, neuropsychological tests, demographics, level of independence in flexibility, working memory and focused attention; and (3) no ADLs as measured using the FIM and response to GNPT tasks: expressive executive deficits. Clusters did not differ in clinical or Cluster 1 (N = 254. 22.9%) the mean age of participants in demographical variables. Cluster 1 at the moment of neuropsychological assessment was Melendez-Moral and colleagues (57) applied CA to address 47 years, 72.8% were males, 68.5% patients with TBI and 22% heterogeneity in mild cognitive impairment (N = 30) using with stroke, 80.7% with <12 years of education. Cluster 2 (N some items of the Test Barcelona, the Wechsler Memory = 376, 33.9%) mean age 54 years, 66.8% males, 53.5% patients Scale III and the TAVEC (Test de Aprendizaje Verbal España- with stroke and 27% other acquired brain injury, 74.7% with <12 Complutense). Three clusters emerged. Cluster 1 corresponded years of education. Cluster 3 (N = 477, 43.2%) mean age 33 years, to the participants with low scores both in memory tests and in 74.8% males, 83% patients with traumatic brain injury and 14% other cognitive domains, Cluster 2 included those subjects who other acquired brain injury, 83.2% with <12 years of education. showed low scores in memory tests, and Cluster 3 included those The FIM is currently considered as the most widely used who did not show low scores in none of the evaluated tests. measure to describe the degree of impairment in activities of In relation to patients with stroke, CA has been mostly daily living in clinical practice (64). Based on Rasch analysis applied to identify communication patterns, in patients with FIM motor scores were categorized in previous research (65) aphasia, therefore using tests not included in Table 1. For into three levels: good, fair, and poor outcomes. A “good” example, Ferré and colleagues (58) analyzed right brain damaged outcome was defined as a patient achieving a FIM motor score individuals (N = 71) from 3 nationalities (Canadians, Brazilians, of 65 or above. With a score of 65, patients usually require Argentinians). CA led to five distinct clinical profiles of either supervision or minimal assistance with mobility and self- communication impairment. care, indicating that the patient’ physical care requirements for Similarly, Akinina et al. (59) recently applied partitioning daily activities are minimal. Scores above 46 indicate some clustering to explore patterns of impairment in verb and improvement (“fair” outcome) and scores under 46 indicate a sentence processing in a sample (N = 54) of people with large physical burden of care (“poor” outcome). As shown in aphasia. The analysis yielded a two clusters solution. The main Table 7, mean motor FIM for patients in Cluster 1 was 48.2(27.2), Frontiers in Neurology | www.frontiersin.org 9 August 2021 | Volume 12 | Article 701946 García-Rudolph et al. Clustering Neuropsychological Assessments motor FIM boxplots are presented in Supplementary Figure 8, customized basis according to an individual’s performance; (2) median and IQR values were 44.0 (22.0–74.0) suggesting a large their easy access on home computers and hand held devices; (3) burden of care (poor outcome). Meanwhile Cluster 2 and 3 their cost-effectiveness compared with paid-person support; and median and IQR values were 55.0 (31.0–76.0) and 59.0 (30.0– (4) most individuals’ familiarity with—and enjoyment using— 80.0) respectively, indicating fair outcome, close to minimal low technology computers and handheld devices. physical care requirements for daily activities. Table 7 shows no A recent systematic review (71) including 28 studies of the use significant differences between Cluster 2 vs Cluster 3 (p = 0.621) of computerized treatment as a rehabilitation tool for attention as also detailed in Supplementary Figure 8. and executive function in adults (aged 18 years or older) who In relation to level of performance in neuropsychological suffered TBI, stroke or other acquired brain injuries. In 23 tests, Tables 4–6 support the differences observed in the FIM. studies, significant improvements in attention and executive According to different authors (30), TMT is one of the most function subsequent to computerized treatment were reported; widely used instruments in neuropsychological assessment. It in the remaining 5, promising trends were observed. has been used to test speed of processing, sequence alternation, In terms of post-stroke cognitive improvement, the cognitive flexibility, visual search, motor performance, and effectiveness of computerized cognitive training in some executive functioning (30). Besides, a positive linear relation specific aspects of cognition, such as memory and executive between brain-injury severity and TMT performance has also functions, has been suggested in recent systematic reviews and been previously reported (66). Nevertheless, normative TMT clinical guidelines (72–74). values for Spanish population with ABI are scarce. According to A systematic review and meta-analysis (75) based on a set of normative values (30) specifically developed for Spanish randomized controlled trials from the last 10 years (2010– healthy middle-aged group (25–54 years; education: 0–12 years) 2020) was recently published to identify the effect of computer- mean TMT-A was 33.04 (7.89) and mean TMT-B was 71.5 based training compared to routine methods on post-stroke (31.07). For Spanish population with TBI, mean age 35.26(12.88), cognitive rehabilitation. Ten out of 201 studies were included education: 0–11 years, mean TMT-A was 82.76(52) and mean in the systematic review, with a total of 600 stroke survivors. TMT-B was 218 [155]. Therefore, as presented in Table 4, mean The authors did not obtain supporting evidence to identify TMT-A (T5) values were 68.1 (42.9) and 75.4 (48.6) for Cluster 2 the superiority of computer-based cognitive training for post- and Cluster 3 respectively, in both cases lower than normative stroke cognitive impairment recovery compared to traditional values for Spanish population with TBI. Meanwhile the mean interventions. However, they did not find any evidence for its side value reported in Table 4 for T5 in Cluster 1 was 250.1 (80.8). effects on cognitive rehabilitation. In addition, the advantages of An important difference when comparing Cluster 2 and convenience and time savings [e.g., (76)] with computer-based Cluster 3 in relation to TMT-B can be related to the 180 s cut-off cognitive training cannot be neglected. The authors recommend value used (along with several other tests) for assessing fitness to to conduct more high-quality studies focusing on different illness drive. It is important to remark that car driving is a complex task phases and various types of intervention software in order that requires the successful integration of perceptual, physical, to improve the meta-analysis and to explore the influence of cognitive, and emotional systems (67), beyond the obtained score computer-based cognitive rehabilitation by subgroup analysis. in TMT-B. In this context, our results support the execution of subgroup A systematic review of the evidence for TMT-B cut-off scores analysis using a set of publicly available R libraries (R-3.5.1) in assessing fitness-to-drive (68) concluded that there is informed within a web-based cognitive platform (the GNPT) and extend support for the 180 s TMT-B cut-off. In our case, this cut-off value the application of CA techniques to mixed data variables from sets a clear difference between Cluster 2 and Cluster 3. The mean baseline neuropsychological assessments of patients with ABI. TMT-B score in Cluster 2 of 142.5 (81.8) with median and IQR Such techniques and R implementations can be used within 122.0 (92.2–170.0) was below the 180.0 s cut-off. Meanwhile for other state-of-the-art computer-based cognitive rehabilitation Cluster 3 the mean TMT-B score of 209.6 (147.9) with median platforms for subgroup analysis involving mixed data variables. and IQR 157.0 (107.0–255.5) was above the 180.0 s cut-off. Furthermore, as presented in the Introduction section of this We are not suggesting that patients in Cluster 2 would be able work, the Intelligent Therapy Assistant (ITA) (17) is already to successfully return to drive, neither that patients in Cluster 3 integrated in the GNPT platform. The ITA provides therapists would not, we are mentioning the 180 s cut-off value with the with a recommended schedule of cognitive tasks to be executed only purpose of remarking the differences between Cluster 2 and by each patient during a given period of time. In order to Cluster 3 in a relevant test such as TMT-B. propose such schedule of tasks, the ITA takes as starting point The level of interest in computerized cognitive training is a set of patient’s cognitive profiles, obtained using CA from the growing more rapidly than other areas of rehabilitation aimed at baseline neuropsychological assessment (17). Our results provide healthy aging, possibly due to the increasing evidence of efficacy, an alternative (straightforward and sound) initial step for the sophistication of delivery systems, and accessibility of these ITA, leaving room for a future comparative study involving the systems across different platforms. The scientific community actual CA approach with the results of this work. interest is also shown by the increasing number of related PubMed publications (69). LIMITATIONS OF THIS STUDY Recently reported (70) advantages of computerized cognitive programs include: (1) their programed self-sufficiency giving Several limitations to our study need to be remarked. First, feedback, adjusting task difficulty and changing tasks on a we conducted a single-center study; an advantage of this is Frontiers in Neurology | www.frontiersin.org 10 August 2021 | Volume 12 | Article 701946 García-Rudolph et al. Clustering Neuropsychological Assessments that data were reported by clinicians, with specific training in years and could be regarded as outdated. Nevertheless, cluster the applied neuropsychological rehabilitation procedures, and analysis has been extensively applied using such tests as input all patients were managed under the same ABI rehabilitation in recent publications (52–63) and those tests are widely used protocols. Nevertheless, the GNPT platform is already integrated as standard tools for neuropsychological evaluations in adults into the clinical practice of several centers specialized in ABI nowadays as recently reported (81). care. Their patients were not included in this analysis, leaving space for future work. An important aspect to address in such CONCLUSIONS future multicenter work, would be in relation to the external validation assessments (for example the FIM used in this study) CA techniques implemented using a set of publicly available which should be common to all participating centers. R libraries were applied in this work to mixed (numerical Second, the health area studied belongs mainly to the Catalan and categorical) variables from baseline neuropsychological Health Service, (with 75.5% of the 1,107 included patients from assessments of patients with ABI, allowing the identification Catalonia and 24,5% from the rest of Spain). Furthermore, of three clinically sound and meaningful patients’ profiles. considering only participants from Catalonia, 79.5% of them The application of several state-of-the-art indexes confirmed were from Barcelona, 10.1% from Girona, 5.4% from Lleida the clusters’ strong internal validity and stability. The external and 5.1% from Tarragona). Therefore, from the total 1,107 validity was also confirmed, considering a relevant aspect of ABI included patients 60.1% were from Barcelona, that may be prone rehabilitation such as functional independence in activities of to selection bias. Nevertheless, we included standardized and daily life and using all cognitive rehabilitation tasks executed by widely applied assessments tools, extensively applied in related the patients included in the study along their whole rehabilitation clinical centers. process, in a web-based cognitive rehabilitation platform. The Third, our analysis lacked computerized tomography or applied CA techniques and R implementations could eventually magnetic resonance imaging examinations that describe the be used within other state-of-the-art computer-based cognitive presence of contusion, hematoma, hemorrhage, ischemia, or rehabilitation tools for subgroup analysis involving mixed data other signs of parenchymal lesion on frontal, temporal, parietal, variables, given the growing level of interest in computerized occipital, and cerebellar lobes or diffuse axonal injury. cognitive rehabilitation treatments, increasingly integrated into Fourth, k-prototypes implements a partitional clustering clinical practice. strategy, the most studied research theme in mixed data clustering as recently reported (42). Nevertheless, other DATA AVAILABILITY STATEMENT available approaches such as hierarchical or model- based, can be considered. In this work we included as The datasets generated for this study will be made available upon Supplementary Material_II the application of a popular reasonable request to the corresponding author of the article. hierarchical strategy for mixed data types, the Partitioning Around Medoids (PAM) (77) with Gower’s dissimilarity, as ETHICS STATEMENT implemented in the functions daisy () of the R package cluster (78) and gowdis () of the fd R package (79). As shown in A specific written informed consent was not required for Supplementary Material_II, the categorical variables seem to participants to be included in this study, in accordance mask the other variables. For example, when including the injury with the local legislation and institutional requirements. categorical variable, the optimal solution yields to three clusters Nevertheless at admission participants provide written with all patients with TBI in one cluster (and only patients with informed consent to be included in research studies TBI), all patients with stroke (and only patients with stroke) addressed by the Institut Guttmann hospital. The authors in the second cluster and all patients with other ABI (and only confirm that this study is compliant with the Helsinki patients with other ABI) in the third cluster. Therefore, this Declaration of 1975, as revised in 2008 and it was approach requires further analysis leaving room for future work. approved by the Ethics Committee of Clinical Research of Fifth, our analysis did not include indicators of mental Institut Guttmann. health or other comorbidities. Still, other medical comorbidities may begin months or years following injury in comparison to AUTHOR CONTRIBUTIONS uninjured control groups. For example, previous studies have reported that individuals with TBI have more than twice the AG-R pre-processed the data, conducted data analysis, prepared rates of pain, growth hormone deficiency, insomnia, fatigue, the figures, drafted the initial manuscript, and revised the new-onset stroke, urinary incontinence, and epilepsy (80). manuscript. AG-M designed the study, pre-processed the data, Therefore, we aim to include comorbidity analysis in future interpreted the data, and revised the manuscript. JMT designed research studies. the study, revised the manuscript, and interpreted the data. EO Sixth, a final point which is important to clarify is in relation designed the study, conducted data analysis, interpreted the data, to the actual tests used in this study. WAIS-III, TMT, RAVLT, and revised the manuscript. DF and VIM acquired funding for WCST, STROOP or Barcelona Test date back over more than 30 the project leading to this publication and revised the manuscript. Frontiers in Neurology | www.frontiersin.org 11 August 2021 | Volume 12 | Article 701946 García-Rudolph et al. Clustering Neuropsychological Assessments MB interpreted the data and revised the manuscript. All authors of Life–European Union’s Horizon 2020 research and innovation have contributed to manuscript revision, read, and approved the programme under grant agreement No. 777107. submitted version. SUPPLEMENTARY MATERIAL FUNDING The Supplementary Material for this article can be found This research was partially funded by PRECISE4Q Personalised online at: https://www.frontiersin.org/articles/10.3389/fneur. Medicine by Predictive Modelling in Stroke for better Quality 2021.701946/full#supplementary-material REFERENCES 16. Gil-Pagés M, Solana J, Sánchez-Carrión R, Tormos JM, Enseñat- Cantallops A, García-Molina A. A customized home-based computerized 1. Cappa SF, Benke T, Clarke S, Rossi B, Stemmer B, van Heugten CM. Task force cognitive rehabilitation platform for patients with chronic-stage on cognitive rehabilitation; european federation of neurological societies. stroke: study protocol for a randomized controlled trial. Trials. (2018) EFNS guidelines on cognitive rehabilitation: report of an EFNS task force. Eur 19:191. doi: 10.1186/s13063-018-2577-8 J Neurol. (2005) 12:665–80. doi: 10.1111/j.1468-1331.2005.01330.x 17. Solana J, Cáceres C, García-Molina A, Chausa P, Opisso E, Roig-Rovira 2. Bayley MT, Tate R, Douglas JM, Turkstra LS, Ponsford J, Stergiou-Kita M, et al. T, et al. Intelligent Therapy Assistant (ITA) for cognitive rehabilitation in INCOG expert panel. INCOG guidelines for cognitive rehabilitation following patients with acquired brain injury. BMC Med Inform Decis Mak. (2014) traumatic brain injury: methods and overview. J Head Trauma Rehabil. (2014) 14:58. doi: 10.1186/1472-6947-14-58 29:290–306. doi: 10.1097/HTR.0000000000000070 18. Sigmundsdottir L, Longley WA, Tate RL. Computerised cognitive 3. Gates NJ, Sachdev PS, Fiatarone Singh MA, Valenzuela M. Cognitive and training in acquired brain injury: A systematic review of outcomes memory training in adults at risk of dementia: a systematic review. BMC using the International Classification of Functioning (ICF). Geriatr. (2011) 11:55. doi: 10.1186/1471-2318-11-55 Neuropsychol Rehabil. (2016) 26:673–741. doi: 10.1080/09602011.2016.114 4. Cha YJ, Kim H. Effect of computer-based cognitive rehabilitation (CBCR) 0657 for people with stroke: a systematic review and meta-analysis. Neuro 19. IG – JCI. https://www.guttmann.com/en/noticias/institut-guttmann- Rehabilitation. (2013) 32:359–68. doi: 10.3233/NRE-130856 successfully-passed-4th-re-accreditation-joint-commission-international 5. Kueider AM, Parisi JM, Gross AL, Rebok GW. Computerized cognitive (accessed April 2021). training with older adults: a systematic review. PLoS ONE. (2012) 20. STROBE. The Strengthening the Reporting of Observational Studies in 7:e40588. doi: 10.1371/journal.pone.0040588 Epidemiology (STROBE) Statement: guidelines for reporting observational 6. Harvey PD, McGurk SR, Mahncke H, Wykes T. Controversies in studies. https://www.equator-network.org/reporting-guidelines/strobe/ computerized cognitive training. Biol Psychiatry Cogn Neurosci Neuroimaging. (accessed April 2021) (2018) 3:907–15. doi: 10.1016/j.bpsc.2018.06.008 21. Peña-Casanova J, Jarne Esparcia AJ, Guardia Olmos. Programa 7. Whitmer AJ, Gotlib IH. Switching and backward inhibition in major integrado de exploración neuropsicológica — test barcelona: validez depressive disorder: the role of rumination. J Abnorm Psychol. (2012) 121:570– de contenidos. Revista de Logopedia, Foniatría y Audiología. (1991) 8. doi: 10.1037/a0027474 11:96–107. doi: 10.1016/S0214-4603(91)75507-1 8. Preiss M, Shatil E, Cermáková R, Cimermanová D, Ram I. 22. Quintana M, Peña-Casanova J, Sánchez-Benavides G, Langohr K, Manero Personalized cognitive training in unipolar and bipolar disorder: RM, Aguilar M, et al. Neuronorma Study Team. Spanish multicenter a study of cognitive functioning. Front Hum Neurosci. (2013) normative studies (Neuronorma project): norms for the abbreviated 7:108. doi: 10.3389/fnhum.2013.00108 Barcelona Test. Arch Clin Neuropsychol. (2011) 26:144–57. 9. Allen DN, Goldstein G. Cluster Analysis in Neuropsychological Research: 23. Wechsler D. Wechsler adult intelligence scale-III. San Antonio: Psychological Recent Applications. New York, NY: Springer Science & Business Media Corp (1997) doi: 10.1037/t49755-000 (2013). doi: 10.1007/978-1-4614-6744-1 24. Reitan RM, Wolfson D. The Halstead–Reitan Neuropsycholgical Test 10. Goldstein, G. Application of cluster analysis to investigate neuropsychological Battery: Therapy and clinicalinterpretation. Tucson, AZ: Neuropsychological heterogeneity in psychiatric and neurological patients. In: Allen DN, Press (1985). Goldstein G, editors. Cluster analysis in neuropsychological research: 25. Stroop J. R. Studies of interference in serial verbal reactions. J. Exp. Psychol. Recent applications. Springer Science + Business Media (2013). p. 37– (193:18, 643–662. doi: 10.1037/h0054651 70. doi: 10.1007/978-1-4614-6744-1_3 26. Scarpina F, Tagini S. The Stroop Color and Word Test. Front Psychol. (2017) 11. Oberholzer M, Müri RM. Neurorehabilitation of Traumatic 8:557. doi: 10.3389/fpsyg.2017.00557 Brain Injury (TBI): A Clinical Review. Med Sci (Basel). (2019) 27. Ryan JJ, Lopez SJ. Wechsler Adult Intelligence Scale-III. In: Dorfman 7:47. doi: 10.3390/medsci7030047 WI, Hersen M, editors. Understanding Psychological Assessment. 12. Arbelaitz O, Gurrutxaga I, Muguerza J, Pérez JM, Perona I. An extensive Perspectives on Individual Differences. Boston, MA: Springer. comparative study of cluster validity indices. Pattern Recognition. (2013) (2001). doi: 10.1007/978-1-4615-1185-4_2 46:243–56 doi: 10.1016/j.patcog.2012.07.021 28. Schmid M. Rey Auditory and Verbal Learning Test: A Handbook. Los Angeles: 13. Polinder S, Cnossen MC, Real RGL, Covic A, Gorbunova A, Western Psychological Services. (1996). Voormolen DC, et al. A multidimensional approach to post-concussion 29. D’Souza A, Mollayeva S, Pacheco N,Javed F, Colantonio A, Mollayeva T. symptoms in mild traumatic brain injury. Front Neurol. (2018) Measuring changeover time: a systematic review of evaluative measures 9:1113. doi: 10.3389/fneur.2018.01113 of cognitive functioning in traumatic brain injury. Front Neurol. (2019) 14. Pérez LM, Inzitari M, Quinn TJ, Montaner J, Gavaldà R, Duarte E, 10:353. doi: 10.3389/fneur.2019.00353 et al. Rehabilitation Profiles of Older Adult Stroke Survivors Admitted 30. Periáñez JA, Ríos-Lago M, Rodríguez-Sánchez JM, Adrover-Roig D, Sánchez- to Intermediate Care Units: A Multi-Centre Study. PLoS ONE. (2016) Cubillo I, Crespo-Facorro B, et al. Trail Making Test in traumatic brain injury, 11:e0166304. doi: 10.1371/journal.pone.0166304 schizophrenia, and normal ageing: sample comparisons and normative data. 15. Garcia-Rudolph A, Garcia-Molina A, Opisso E, Tormos Muñoz J. Arch Clin Neuropsychol. (2007) 22:433–47. Personalized web-based cognitive rehabilitation treatments for patients 31. Grant DA, Berg EA. A behavioural analysis of degree of reinforcement and with traumatic brain injury: cluster analysis. JMIR Med Inform. (2020) ease of shifting to newresponses in a Weigl-type card sorting problem. J Exp 8:e16077. doi: 10.2196/16077 Psychol. (1948) 38, 404–411 doi: 10.1037/h0059831 Frontiers in Neurology | www.frontiersin.org 12 August 2021 | Volume 12 | Article 701946 García-Rudolph et al. Clustering Neuropsychological Assessments 32. Heaton RK, Chelune GJ, Talley JL, Kay GC, Curtiss G. Wisconsin card sorting 55. Sherer M, Sander AM, Nick TG, Melguizo MS, Tulsky DS, Kisala P, et al. test Manual. Odessa, FL: Psychological Assessment Resources (1993). Key dimensions of impairment, self-report, and environmental supports 33. Ben-David BM, Nguyen LL, van Lieshout PH. Stroop effects in persons in persons with traumatic brain injury. Rehabil Psychol. (2015) 60:138– with traumatic brain injury: selective attention, speed of processing, 46. doi: 10.1037/rep0000030 or color-naming? A meta-analysis. J Int Neuropsychol Soc. (2011) 56. Zimmermann N, Pereira N, Hermes-Pereira A, Holz M, Joanette Y, Fonseca 17:354–63. RP. Executive functions profiles in traumatic brain injury adults: Implications 34. Solana J, Cáceres C, García-Molina A, Opisso E, Roig T, Tormos JM, for rehabilitation studies. Brain Inj. (2015) 29:1071–81. et al. Improving brain injury cognitive rehabilitation by personalized 57. Meléndez Moral JC, Sanz-Álvarez T, Navarro-Pardo E. Mild cognitive telerehabilitation services: Guttmann neuropersonal trainer. IEEE J Biomed impairment: classification method and procedure. Anales de Psicología. (2012) Health Inform. (2015) 19:124–31. 28. doi: 10.6018/analesps.28.2.148891 35. R version. https://cran.r-project.org/bin/windows/base/ (accessed April 2021) 58. Ferré P, Fonseca RP, Ska B, Joanette Y. Communicative clusters after a right- 36. Corrplot R package. https://cran.r-project.org/web/packages/corrplot/ hemisphere stroke: are there universal clinical profiles? Folia Phoniatr Logop. corrplot.pdf (accessed April 2021) (2012) 64:199–207. 37. Z-normalizacion. http://finzi.psych.upenn.edu/R/library/dprep/html/znorm. 59. Akinina YO, Buivolova O, Soloukhina A, Artemova A, Zyryanov, html (accessed April 2021) Bastiaanse R. Prevalence of verb and sentence impairment in aphasia 38. Base R package. https://www.rdocumentation.org/packages/base/versions/3. as demonstrated by cluster analysis. Aphasiology. Routledge (2020), 6.2/topics/scale (accessed April 2021) p. 1–29. doi: 10.1080/02687038.2020.1812045 39. MacQueen. Some methods for classification and analysis of multivariate 60. Malathi M, Sinthia P. MRI brain tumour segmentation using hybrid clustering observations. In: L. Le Camand J. Neyman, editors, Proc. 5th Berkeley Symp. and classification by back propagation algorithm. Asian Pac J Cancer Prev. Math Stat and Prob. (1967) p 281 – 297. (2018) 19:3257–63. doi: 10.31557/APJCP.2018.19.11.3257 40. Szepannek G. clustMixType: user-friendly clustering of mixed-type data in R. 61. Sişik F, Sert E. Brain tumor segmentation approach based on the R J. (2018) 10:200. doi: 10.32614/RJ-2018-048 extreme learning machine and significantly fast and robust fuzzy C-means 41. Huang Z. Extensions to the k-means algorithm for clustering large clustering algorithms running on Raspberry Pi hardware. Med Hypotheses. data sets with categorical variables. Data Min Knowl Discov. 2:283–304, (2020) 136:109507. doi: 10.1016/j.mehy.2019.109507 1998. doi: 10.1023/A:1009769707641 62. Lemasson B, Collomb N, Arnaud A, Luc Barbier E,Forbes F. Monitoring 42. Ahmad A, Khan S. Survey of State-of-the-Art Mixed glioma heterogeneity during tumor growth using clustering analysis of Data Clustering Algorithms. IEEE Access. (2019) 7:31883– multiparametric MRI data. ISMRM International Society for Magnetic 31902. doi: 10.1109/ACCESS.2019.2903568 Resonance in Medicine. Honolulu, United States (2017). 43. Aschenbruck R, Szepannek G. Cluster Validation for Mixed-Type Data. 63. Hiratsuka T, Arakawa Y, Yajima Y, Kakimoto Y, Shima K, Yamazaki Archives of Data Science, Vol. 6. KIT Scientific Publishing (2020). Y, et al. Hierarchical cluster and region of interest analyses based on doi: 10.5445/KSP/1000098011/02 mass spectrometry imaging of human brain Tumours. Sci Rep. (2020) 44. Hennig, C. Cluster-wise assessment of cluster stability. Computational 10:5757. doi: 10.1038/s41598-020-62176-8 Statistics and Data Analysis. (2007) 52:258–71. doi: 10.1016/j.csda.2006.11.025 64. Scrutinio D, Battista P, Guida P, Lanzillo B, Tortelli R. Sex differences in 45. FPC R package https://cran.r-project.org/web/packages/fpc/fpc.pdf (accessed long-term mortality and functional outcome after rehabilitation in patients April 2021) with severe stroke. Front Neurol. (2020) 11:84. doi: 10.3389/fneur.2020. 46. Hennig C, Meila M, Murtagh F, Rocci R. Handbook of Cluster Analysis. 00084 Chapman and Hall/CRC (2020). 65. Brock KA, Vale SJ, Cotton SM. The effect of the introduction 47. Giles GM, Clark-Wilson J, Baxter DM, Tasker R, Holloway M, Seymour S. The of a case-mix-based funding model of rehabilitation for severe interrelationship of functional skills in individuals living in the community, stroke: an Australian experience. Arch Phys Med Rehabil. (2007) following moderate to severe traumatic brain injury. Brain Inj. (2019) 33:129– 88:827–32. doi: 10.1016/j.apmr.2007.04.001 36. 66. Lange RT, Iverson GL, Zakrzewski MJ, Ethel-King PE, Franzen MD. 48. Linacre JM, Heinemann JW, Wright BD, Granger CV, Hamilton BB. The Interpreting the trail making test following traumatic brain injury: structure and stability of the functional independence measure. Arch Phys Comparison of traditional time scores and derived indices. J Clin Exp Med Rehabil. (1994) 75:127–32. doi: 10.1016/0003-9993(94)90384-0 Neuropsychol. (2005) 27:897–906. doi: 10.1080/1380339049091290 49. Mattei TA, Teasdale GM. The Story of the Development and Adoption 67. Schultheis MT, Whipple E. Driving after traumatic brain injury: evaluation of the Glasgow Coma Scale: Part I, The Early Years. World Neurosurg. and rehabilitation interventions. Curr Phys Med Rehabil Rep. (2014) 2:176–83. (2020) 134:311–22. 68. Roy M, Molnar F. Systematic review of the evidence for Trails B cut-off scores 50. Gocan S, Fisher A. Neurological assessment by nurses using the National in assessing fitness-to-drive. Can Geriatr J. (2013) 16:120–42. Institutes of Health Stroke Scale: implementation of best practice guidelines. 69. Harvey PD, McGurk SR, Mahncke H, Wykes T. Controversies in Can J Neurosci Nurs. (2008) 30:31–42. computerized cognitive training. Biol Psychiatry Cogn Neurosci Neuroimaging. 51. Hamaoka K, Maeda R, Okabayashi M, Sugimoto A, Yamakawa T, Yamanaka (2018) 3:907–15. S, et al. (2014). Functional independence measure reference values for the 70. O’Neil-Pirozzi TM, Hsu H. Feasibility and benefits of computerized cognitive discharge to home of stroke patients: functional independence measure. exercise to adults with chronic moderate-to-severe cognitive impairments J-Stage. (2014) 29:933–7. doi: 10.1589/rika.29.933 following an acquired brain injury: A pilot study. Brain Inj. (2016) 30:1617–25. 52. Thaler NS, Linck JF, Heyanka DJ, Pastorek NJ, Miller B, Romesser J, 71. Bogdanova Y, Yee MK, Ho VT, Cicerone KD. Computerized cognitive et al. Heterogeneity in Trail Making Test performance in OEF/OIF/OND rehabilitation of attention and executive function in acquired brain veterans with mild traumatic brain injury. Arch Clin Neuropsychol. (2013) 28: injury: a systematic review. J Head Trauma Rehabil. (2016) 31:419– 798–807. doi: 10.1093/arclin/act080 33. doi: 10.1097/HTR.0000000000000203 53. Harman-Smith YE, Mathias JL, Bowden SC, Rosenfeld JV, Bigler ED. 72. Nousia A, Martzoukou M, Siokas V, Aretouli E, Aloizou AM, Folia V, et al. Wechsler Adult Intelligence Scale-Third Edition profiles and their Beneficial effect of computer-based multidomain cognitive training in patients relationship to self-reported outcome following traumatic brain injury. J with mild cognitive impairment. Applied Neuropsychology: Adult. (2019),1– Clin Exp Neuropsychol. (2013) 35:785–98. 10. doi: 10.1080/23279095.2019.1692842 54. Sherer M, Ponsford J, Hicks A, Leon-Novelo L, Ngan E, Sander AM. Cross- 73. Svaerke K, Niemeijer M, Mogensen J, Christensen H. The effects of Validation of a Classification System for Persons With Traumatic Brain computerbased cognitive rehabilitation in patients with visuospatial neglect Injury in the Posthospital Period. J Head Trauma Rehabil. (2017) 32(5):E17– following stroke: A systematic review. Topics in Stroke Rehabilitation. (2018) 25. doi: 10.1097/HTR.0000000000000290 26:214–225. doi: 10.1080/10749357.2018.1556963 Frontiers in Neurology | www.frontiersin.org 13 August 2021 | Volume 12 | Article 701946 García-Rudolph et al. Clustering Neuropsychological Assessments 74. van de Ven RM, Murre JMJ, Veltman DJ, Schmand BA. Computer-based Conflict of Interest: AG-R, AG-M, JMT, EO and MB work at Institut cognitive training for executive functions after stroke: A systematic review. Guttmann, Hospital de Neurorehabilitació, proprietary of the Guttmann, Front Human Neurosci. (2016) 10:150. doi: 10.3389/fnhum.2016.00150 NeuroPersonalTrainer R platform. VIM reported receiving personal fees from 75. Ye M, Zhao B, Liu Z, Weng Y, Zhou L. Effectiveness of computer- ai4medicine outside the submitted work. There is no connection, commercial based training on post-stroke cognitive rehabilitation: A systematic exploitation, transfer or association between the projects of ai4medicine and the review and meta-analysis. Neuropsychol Rehabil. (2020) 23:1–17. results presented in this work. doi: 10.1080/09602011.2020.1831555 76. Nyberg CK, Nordvik JE, Becker F, Rohani DA, Sederevicius D, Fjell AM, et al. The remaining author declares that the research was conducted in the absence of A longitudinal study of computerized cognitive training in stroke patients - any commercial or financial relationships that could be construed as a potential effects on cognitive function and white matter. Topics in Stroke Rehabilitation. conflict of interest. (2018). 25:241–7. doi: 10.1080/10749357.2018.1443570 77. Van de Velden M, Iodice D’Enza A, Markos A. Distance-based clustering of Publisher’s Note: All claims expressed in this article are solely those of the authors mixed data WIREs. Comput Stat. (2019). 11:e1456. doi: 10.1002/wics.1456 and do not necessarily represent those of their affiliated organizations, or those of 78. Maechler M, Rousseeuw P, Struyf A, Hubert M, Hornik K. cluster: Cluster Analysis Basics and Extensions [Computer software manual]. (2018) Available the publisher, the editors and the reviewers. Any product that may be evaluated in online at: https://CRAN.R-project.org/package=cluster this article, or claim that may be made by its manufacturer, is not guaranteed or 79. Lalibert’e E, Legendre P, Shipley B. FD: measuring functional diversity from endorsed by the publisher. multiple traits, and other tools for functional ecology [Computer software manual] (2014). Available online at: https://CRAN.R-project.org/package=FD Copyright © 2021 García-Rudolph, García-Molina, Opisso, Tormos, Madai, Frey 80. Hammond F, Corrigan J, Ketchum JM, Malec JF, Dams-O’Connor K, and Bernabeu. This is an open-access article distributed under the terms of Hart T, et al. Prevalence of medical and psychiatric comorbidities the Creative Commons Attribution License (CC BY). The use, distribution or following traumatic brain injury. J Head Trauma Rehabil. (2019) 34:E1– reproduction in other forums is permitted, provided the original author(s) and the E10. doi: 10.1097/HTR.0000000000000465 copyright owner(s) are credited and that the original publication in this journal 81. Schroeder RW, Martin PK, Walling A. Neuropsychological Evaluations in is cited, in accordance with accepted academic practice. No use, distribution or Adults. Am Fam Physician. (2019) 99:101–108. reproduction is permitted which does not comply with these terms. Frontiers in Neurology | www.frontiersin.org 14 August 2021 | Volume 12 | Article 701946