ORIGINAL RESEARCH published: 15 October 2021 doi: 10.3389/fdgth.2021.750661 DCTclock: Clinically-Interpretable and Automated Artificial Intelligence Analysis of Drawing Behavior for Capturing Cognition William Souillard-Mandar 1,2,3*, Dana Penney 1,4 , Braydon Schaible 1 , Alvaro Pascual-Leone 2,5,6,7 , Rhoda Au 8 and Randall Davis 1,3 1 Digital Cognition Technologies, Boston, MA, United States, 2 Linus Health, Boston, MA, United States, 3 Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, MA, United States, 4 Department of Neurology, Lahey Hospital and Medical Center, Boston, MA, United States, 5 Deanna and Sidney Wolk Center for Memory Health, Arthur and Hinda Marcus Institute for Aging Research and Hebrew SeniorLife, Boston, MA, United States, 6 Department of Neurology, Harvard Medical School, Boston, MA, United States, 7 Guttmann Brain Health Institute, Institut Guttmann de Neurorehabilitació, Universitat Autonoma de Barcelona, Barcelona, Spain, 8 Departments of Anatomy & Neurobiology, and Neurology and Framingham Heart Study, Boston University School of Medicine, and Department of Epidemiology, Boston University School of Public Health, Boston, MA, United States Developing tools for efficiently measuring cognitive change specifically and brain health generally—whether for clinical use or as endpoints in clinical trials—is a major Edited by: challenge, particularly for conditions such as Alzheimer’s disease. Technology such Manousos A. Klados, International Faculty of the University as connected devices and advances in artificial intelligence offer the possibility of of Sheffield, Greece creating and deploying clinical-grade tools with high sensitivity, rapidly, cheaply, and Reviewed by: non-intrusively. Starting from a widely-used paper and pencil cognitive status test—The Ivan Miguel Pires, Universidade da Beira Clock Drawing Test—we combined a digital input device to capture time-stamped Interior, Portugal drawing coordinates with a machine learning analysis of drawing behavior to create Amaia Mendez Zorrilla, DCTclockTM , an automated analysis of nuances in cognitive performance beyond University of Deusto, Spain successful task completion. Development and validation was conducted on a dataset *Correspondence: William Souillard-Mandar of 1,833 presumed cognitively unimpaired and clinically diagnosed cognitively impaired w.souillardmandar@alum.mit.edu individuals with varied neurological conditions. We benchmarked DCTclock against existing clock scoring systems and the Mini-Mental Status Examination, a widely-used Specialty section: This article was submitted to but lengthier cognitive test, and showed that DCTclock offered a significant improvement Connected Health, in the detection of early cognitive impairment and the ability to characterize individuals a section of the journal Frontiers in Digital Health along the Alzheimer’s disease trajectory. This offers an example of a robust framework Received: 30 July 2021 for creating digital biomarkers that can be used clinically and in research for assessing Accepted: 10 September 2021 neurological function. Published: 15 October 2021 Keywords: artificial intelligence, cognition, Alzheimer’s disease, dementia, DCTclock, clock drawing test, behavior Citation: analysis Souillard-Mandar W, Penney D, Schaible B, Pascual-Leone A, Au R and Davis R (2021) DCTclock: Clinically-Interpretable and Automated INTRODUCTION Artificial Intelligence Analysis of Drawing Behavior for Capturing Early detection and diagnosis of cognitive decline is critical to the development and deployment Cognition. of novel therapeutic interventions for patients with dementia due to Alzheimer’s disease Front. Digit. Health 3:750661. (AD) and other neurodegenerative diseases. Clinical detection currently largely depends on doi: 10.3389/fdgth.2021.750661 reported complaints from patients or their family, or impaired performance on standard Frontiers in Digital Health | www.frontiersin.org 1 October 2021 | Volume 3 | Article 750661 Souillard-Mandar et al. DCTclock: AI Analysis of Drawing Behavior cognitive screening tasks such as the Mini Mental State opportunity to capture early change by revealing subtle behaviors Examination (MMSE) (1) or the Montreal Cognitive Assessment that precede impairment such as decision-making latencies, (MoCA) (2). These tests have been shown to be, relatively compensatory strategies, and psycho-motor issues not visible in insensitive to milder forms of impairment and require the final drawing. administration and hand-scoring by trained administrators, To address these issues, a first digital version of the which can lead to subjectivity in both scoring and interpretation. CDT was created by using a digitizing pen to capture the Computerized tests, such as CANTAB Mobile (3) and Cogstate entire drawing process—both spatial and temporal data—while Brief Battery (4) have made significant strides in tackling these keeping the well-known administration procedures for the clock problems by automating administration and scoring processes. drawing test administration standard. The digitizing pen works These tests, however, measure behaviors that are more unnatural as a standard ballpoint, enabling drawing on a paper and and distant from daily living, which, combined with lengthier avoiding the use of a digitizing tablet, which is less natural testing sessions, can limit the broad use of the technology, for the elderly. From a digitally captured CDT performance, especially in older individuals. Cerebrospinal fluid and positron two human raters independently classified the digital pen emission tomography (PET) imaging– gold standard diagnostic coordinates into drawing components (i.e., digits, hands), with indices of earliest stage disease in neurological disorders such as a third reviewer for resolving any differences. This enabled the AD—remain costly and invasive, have suboptimal accuracy in creation and computation of novel drawing measures from these predicting cognitively related impairment, and provide limited classifications, such as between pen stroke latencies and drawing clinical information for addressing patient and family concerns. velocities. Using these novel measurements, classification models Pathology associated with AD and other neurodegenerative were developed for the detection of cognitive impairment, diseases is present for years, often decades, prior to diagnosis, with a focus on neurodegenerative disorders such as AD and and diagnosis at the point of overt clinical manifestations may Parkinson’s disease, showing significant improvements over be too late to meaningfully impact outcome and prognosis existing manual scoring systems for the CDT (10). These (5). Thus, there is not only a need for early diagnosis but classification models were then further refined using clinician also for screening and detection of abnormalities in otherwise input to make the classification results more transparent and asymptomatic individuals. To that end, digital markers might interpretable to clinicians, with only a small trade-off in accuracy be ideally suited as rapid, user-friendly, and objective screening (10). Prior research has shown that patients have longer total tools for detecting and tracking subtle cognitive change, and drawing times for completing the CDT task after total knee as differential diagnosis tools that provide clinically useful arthroplasty with general anesthesia (11) and during normal and patient-centered information to inform treatment. These aging (12), and that specific process latencies during the drawing characteristics would enable the idea of a new “vital sign” for can reveal deficits in processing speed and decision making in the brain. subjects with Multiple Sclerosis (13). Previous work has also As a simple paper and pencil test used for over 50 years, shown that the order of clock component placement in non- the Clock Drawing Test (CDT) is quick and easy to administer, demented depressed older adults is associated with cognitive and non-invasive and inexpensive, yet provides valuable clinical brain structural connectome differences (14). information and diagnostic utility (6). The test instructions first Despite promising results, this first version of the digital ask the subject to draw, on a blank sheet of paper, a clock CDT suffered from significant limitations. First, it required showing 10 min after 11 (Command clock), and then asks them lengthy human review of the drawings to classify the pen strokes to copy a pre-drawn clock showing that time (Copy clock). into symbols, necessary to extract clinically meaningful features, Successful completion of the test requires a variety of cognitive rendering the test time-consuming, prone to human error, and functions, including memory (working memory, visual memory, unable to provide immediate, live results to a clinician or patient. semantic memory), executive function, selective and sustained Second, more validation data was required to demonstrate overall attention, visuospatial abilities, auditory comprehension, and test accuracy, benchmark performance against other commonly- motor control (7). It has been a useful screening tool to used tests, and understand clinical benefits. Third, the software differentiate normal elderly individuals from those with cognitive and hardware of the system was not constructed to scale beyond impairment, and has been effective in helping to diagnose research use, and there was a clear need to convert it into an dementia, Parkinson’s disease, and other conditions (8). easily-deployable platform. These issues limited the digital CDT However, the utility of the traditional CDT is reduced by to research use and made it impossible to deploy clinically or a lack of consistency between diverse scoring systems, the at scale. complexity and often imprecise scoring criteria, and inter-rater The present study tackles the above limitations, summarizing unreliability due to the subjective nature of some scoring criteria the development and validation of DCTclock, a fully automated (9). Additionally, clinical assessment generally considers only the analysis of the drawing process and final drawing from the digital final drawing, often with no evaluation of the drawing process. CDT, as a screening tool for cognitive impairment. We describe While more impaired patients may make significant errors in how the algorithms were developed, the resulting clinical scores, their drawings, subtly impaired patients may produce a final and the validation findings based on comparisons between AD drawing that appears normal or near-normal and are thus at and aMCI clinical samples and a community-based normal aging risk for having early cognitive change missed by the clinician. population. A separate validation study of this technology in In contrast, analyzing the entire drawing process offers better pre-clinical AD individuals has already shown that DCTclock Frontiers in Digital Health | www.frontiersin.org 2 October 2021 | Volume 3 | Article 750661 Souillard-Mandar et al. DCTclock: AI Analysis of Drawing Behavior is strongly correlated with tau and amyloid loads on PET (15). a dataset of presumed cognitively unimpaired and clinically DCTclock is FDA-cleared to market as a computerized cognitive diagnosed cognitively impaired individuals, respectively (N = assessment aid, offering a sensitive and scalable early detection 1,833). The combined dataset was randomly divided into training method for cognitive health and dementia. (N = 912) and testing (N = 921) sets, stratified by the presence of cognitive impairment to keep the classes balanced in both MATERIALS AND METHODS datasets. Subjects contained within the training set were used solely for development purposes (i.e., building the machine Compliance With Ethical Regulations learning models), while subjects within the testing set were used All procedures followed the ethical guidelines of the 1975 exclusively for statistical and clinical validation. A subset of Declaration of Helsinki and data was collected under IRB the testing set was created for participants with both DCTclock approved protocols at both sites–Framingham Heart Study and and MMSE administered within 18 months of each other (N Lahey Hospital and Medical Center. Data analysis herein was = 591) to enable the benchmark of these two tests. While conducted under an IRB approved protocol at Digital Cognition this time difference is significant and could lead to changes in Technologies/Linus Health (IRB Tracking Number 20160721). performance, the most rapid change would be expected from cognitive impaired LH participants who had DCTclock and Study Participants MMSE within a 2 week period, while the presumed cognitively Since 2005, 1,560 subjects from an outpatient Neurology unimpaired from FHS would likely fluctuate less rapidly making service at Lahey Hospital and Medical Center (LH) have been the up to 18 months difference more acceptable. Demographics administered DCTclock. Subjects were administered a subset for both the training and testing data sets are summarized of the following neuropsychological tests within 2 weeks of in Table 1. DCTclock: MMSE, MoCA, Repeatable Battery for the Assessment of Neuropsychological Status (RBANS) (16), Halstad-Reitan DCT Platform Trail Making Test (Trails) A and B (17), Hooper Visual The DCT platform consists of the digitizing ballpoint pen Organization Test (HVOT) (18), Boston Naming Test (BNT) (Figure 1A), a computer application that sends encrypted test (19), and Beck Depression Inventory (BDI) (20). All individuals data to remote servers where it is analyzed, and a web portal had documented medical and cognitive status including medical, to view and interact with results (Figure 1D). DCTclock is the neurological, surgical, and psychiatric conditions based on first cognitive test built on top of the DCT platform. The digital consensus diagnosis. Consensus diagnosis was derived from pen functions as an ordinary ballpoint pen while recording imaging, neuropsychological testing, and medical record review, its position on the page with considerable spatial (±0.05 cm) independently of DCTclock and MMSE performance. Three and temporal (12 ms) accuracy. The raw data—a set of time- groups of subjects aged 55+ were selected from this population stamped points—is encrypted by the pen and then transferred based on consensus diagnosis: a general cognitively impaired to a computer that sends the data to DCT servers. The data is group (N = 791), and subsets of individuals diagnosed with securely stored and analysis software is run to analyze the tests. amnestic mild cognitive impairment (aMCI; N = 113) and The test administrators and clinicians may log into a web portal dementia due to probable AD (N = 128). to view the tests and results. Beginning in 2011, DCTclock was administered as part of the standard cognitive assessment battery to participants enrolled DCTclock Algorithm in the Framingham Heart Study (FHS) cognitive aging and Pen Stroke Classification for Drawing Understanding dementia studies, an established longitudinal epidemiological The first step in the analysis consisted of developing an research program. FHS has conducted on-going surveillance for understanding of the drawing by mapping each pen stroke into incident dementia since 1976 (21) and incident stroke since the expected symbols for the representation of a clock—clock face, study inception in 1948 (22). DCTclock was administered to digits, hands—or as unexpected or unnecessary symbols such participants in the Original, Offspring, New Offspring Spouses, as small noise strokes, cross-outs, and overwriting (Figure 1B). and OmniGeneration 1 cohorts (N = 2,309). For this study, We trained a convolutional neural network architecture (23) to participants were excluded if aged below 55 (N = 610), recognize these individual symbols by using our own database and diagnosed with prevalent dementia and/or stroke (N = of labeled symbols from our training set and supplementing 76). Additionally, examiner impression of possible cognitive it with publicly available digits from MNIST (24). We then impairment was used to exclude individuals not deemed to be developed a multi-pass, multi-layer rules algorithm for pen stroke cognitively unimpaired on the basis of clinical impression by the classification into drawing symbols, that iteratively combined pen examining technician, who was not a licensed neuropsychologist strokes based on their spatial and temporal distribution within (N = 581). This formed our cognitively unimpaired sample (N the drawing process and assigned symbol probabilities to each = 1,042). A subset of these individuals were also administered combination using the trained neural net. the MMSE as part of their longitudinal involvement in the study, though unlike at LH, there was on average a multi-year difference Drawing Measurements between DCTclock and MMSE. Using the pen stroke classification output, measurements are For DCTclock development, the baseline DCTclock test of calculated from both the final drawing and the drawing each participant from FHS and LH was combined to form process (Figure 1C). Some measurements derived from the final Frontiers in Digital Health | www.frontiersin.org 3 October 2021 | Volume 3 | Article 750661 Souillard-Mandar et al. DCTclock: AI Analysis of Drawing Behavior TABLE 1 | Subject demographics for training and testing sets. Variable Level Training set Testing set P-value Sample size 912 921 Age [mean (sd)] 69.94 (8.49) 69.88 (8.32) 0.867 Cognitive status [N (%)] Cognitively unimpaired 530 (58.1) 512 (55.6) 0.297 Cognitively impaired 382 (41.9) 409 (44.4) Education [N (%)] College graduate 465 (51.0) 476 (51.7) 0.380 Some college 221 (24.2) 245 (26.6) HS graduate 192 (21.1) 167 (18.1) Less than HS 34 (3.7) 33 (3.6) Gender [N (%)] Female 483 (53.0) 449 (48.8) 0.079 Male 429 (47.0) 472 (51.2) Race [N (%)] Asian 21 (2.3) 21 (2.3) 0.984 Black/African American 17 (1.9) 16 (1.8) White/Caucasian 861 (95.8) 863 (95.9) Ethnicity [N (%)] Hispanic/Latino 13 (1.7) 17 (2.2) 0.568 Non-Hispanic/Latino 763 (98.3) 754 (97.8) MMSE Score (median [IQR]) 28 (25, 29) 28 (25, 29) 0.696 FIGURE 1 | DCTclock automated analysis steps. (A) Digitizing pen and standard test paper to capture timestamped-coordinates of drawing motion. (B) Time-stamped pen stroke coordinates are classified into drawing symbols. (C) Using knowledge from the classified strokes, low-level features are extracted from the drawing process. (D) Low-level features are combined into a set of interpretable Composite Scales and an overall score of test performance. drawing, such as the correct placement of clock components, are through clinician feedback. While this resulted in a small loss of equivalent to measurements used in scoring systems from the classification accuracy, it produced increased transparency (10). CDT; others are novel and include the overall position of the These features, denoted as cognitive features and detailed in drawing on the page, the total number of strokes, and the lengths Table 2, were designed to have stronger face validity by looking of various strokes. Most of the measurements, however, are at the drawing or viewing a replay of the drawing process. These focused on the drawing process and include total time, different measurements were assigned to one of four functional groups types of latencies, pen speeds and oscillations during various Drawing Efficiency, Simple and Complex Motor, Information portions of the drawing process, and measures of efficiency in the Processing, and Spatial reasoning. drawing production. For each functional group and for each of the Command and Copy conditions of the clock separately, a composite score Creating Composite Scales and DCTclock Score for was learnt using Lasso regularized logistic regression to classify Cognitive Impairment Classification the Cognitively Impaired and Cognitively Unimpaired groups Measurements were harmonized to produce a collection of in our training set. These eight scores were then combined meaningful metrics of cognition based on brain-behavior using L1 regularized logistic regression to form a top level principles. In this process, opaque variables were eliminated DCTclock score. Frontiers in Digital Health | www.frontiersin.org 4 October 2021 | Volume 3 | Article 750661 Souillard-Mandar et al. DCTclock: AI Analysis of Drawing Behavior TABLE 2 | Table of DCTclock metrics with definitions and median [IQR]. DCTclock Definition Cognitively unimpaired Cognitively impaired metric DCTclock A number between 0 and 100 that represents a 86.23 [71.79, 93.68] 39.16 [14.27, 60.00] score person’s overall cognitive function as assessed by DCTclock Command clock Copy clock Cognitively Cognitively Cognitively Cognitively unimpaired impaired unimpaired impaired Drawing The efficiency the individual demonstrated during the 0.16 −1.11 0.10 −0.91 efficiency process of drawing each clock. This considers [−0.51, 0.66] [−2.62, −0.06] [−0.41, 0.58] [−2.26, 0.07] metrics such as time spent relative to properties of the drawing including number of pen strokes, stroke length, and size of the drawing. Stroke count The deviation from the expected number of pen 0.00 1.00 0.00 0.00 conformity strokes in the drawing. [0.00, 2.00] [0.00, 5.50] [0.00, 1.00] [0.00, 2.00] Total time The total time spent completing the drawing, 34.37 44.42 26.22 32.12 measured from the first touch of the pen on the [27.48, 42.98] [32.04, 65.01] [22.28, 32.49] [25.41, 43.58] paper to the last pen lift off the paper. Ink length The sum, in millimeters, of the lengths of all pen 580.32 480.08 440.93 392.1 strokes used in the drawing. [470.2, 694.1] [365.47, 637.09] [390.92, 495.32] [341.4, 464.6] Drawing Size The size, in millimeters, of the clock face circle. 80.29 64.06 59.15 50.95 [65.69, 94.16] [46.66, 82.06] [52.92, 67.71] [44.18, 59.52] Drawing A relative measure that combines Ink Length and 16.48 10.70 16.63 12.08 Process Total Time. [13.06, 20.75] [6.93, 15.12] [13.11, 20.76] [8.42, 16.01] Efficiency Noise A measure of the drawing that includes non-standard 0.40 0.41 0.39 0.41 pen strokes, cross outs, and overwriting. [0.40, 0.42] [0.40, 0.44] [0.39, 0.42] [0.39, 0.44] Information The non-motor cognitive functions used during the 0.19 −0.65 0.16 −0.74 processing drawing process. This considers metrics such as [−0.49, 0.75] [−1.67, 0.22] [−0.59, 0.74] [−1.70, 0.19] absolute and relative duration of latencies, number of pauses, and relative time spent thinking vs. actively drawing with pen on the paper. Percent think The percentage of the test time spent “thinking” (i.e., 59.48 62.10 53.70 55.84 time holding the pen off the page but not actively drawing) [54.01, 65.19] [54.78, 69.95] [49.00, 58.12] [49.20, 62.46] measured from the first touch of the pen on the paper to the last pen lift off the paper. Average The average duration of the latencies between each 0.84 1.18 0.62 0.80 latency pen stroke. [0.68, 1.10] [0.82, 1.77] [0.51, 0.77] [0.60, 1.08] Latency The variability in the latencies throughout the drawing 0.99 1.62 0.49 0.79 variability process. [0.68, 1.51] [0.96, 2.62] [0.37, 0.68] [0.52, 1.21] Relative long A measure of the difference between the average 7.26 11.62 4.20 5.98 latency latency and the longer latencies within this drawing. [5.01, 10.43] [6.65, 19.96] [3.19, 5.73] [4.09, 9.32] Long latency The total number of latencies in the drawing that are 1.00 2.00 1.00 2.00 count notably longer than the normative sample standard. [0.00, 1.00] [1.00, 3.00] [0.00, 2.00] [1.00, 4.00] Longest The duration of the longest latency in the drawing. 4.46 6.87 1.98 3.12 latency [2.94, 6.84] [4.16, 11.64] [1.50, 2.83] [2.09, 5.37] Simple and The graphomotor components involved in the 0.24 −1.12 0.14 −0.75 complex process of drawing each clock. This considers [−0.48, 0.82] [−2.20, −0.04] [−0.44, 0.68] [−1.68,0.13] motor metrics such as pen stroke speeds and oscillatory motion and can be helpful in parsing out motor and non-motor cognitive functions. Percent Ink The percentage of the test time spent actively 40.52 37.90 46.30 44.16 Time drawing with the pen on the paper. [34.81, 45.99] [30.05, 45.22] [41.88, 51.00] [37.54, 50.80] Average speed The average speed of the pen for all pen strokes 127.09 81.47 100.37 78.74 used during the drawing of the clock face. [91.95, 170.8] [54.75, 129.92] [73.59, 138.40] [51.16, 113.3] Max speed The maximum speed of the pen on the page during 208.9 158.17 168.30 140.3 the drawing of the clock face. [157.9, 265.2] [104.80, 217.74] [129.06, 229.47] [97.12, 193.2] (Continued) Frontiers in Digital Health | www.frontiersin.org 5 October 2021 | Volume 3 | Article 750661 Souillard-Mandar et al. DCTclock: AI Analysis of Drawing Behavior TABLE 2 | Continued DCTclock Definition Cognitively unimpaired Cognitively impaired metric Command clock Copy clock Cognitively Cognitively Cognitively Cognitively unimpaired impaired unimpaired impaired Initiation speed The speed of the pen when beginning to draw the 109.59 76.18 85.16 68.08 clock face. [76.09, 148.5] [50.09, 111.70] [60.15, 119.44] [43.16, 95.27] Termination The speed of the pen when finishing the clock face. 117.93 72.43 94.97 71.90 speed [83.36, 166.7] [45.29, 116.98] [67.99, 138.3] [43.42, 110.1] Oscillatory A measure of how much the motion of the pen 2.16 3.01 2.38 3.04 motion deviates from a smooth motion [1.67, 2.74] [2.35, 3.94] [1.93, 2.83] [2.47, 3.88] Spatial The spatial abilities demonstrated during the drawing 0.31 −1.65 0.27 −1.58 reasoning process. This considers metrics pertaining to the [−0.39, 0.68] [−3.58, −0.21] [−0.63, 0.80] [−2.81, −0.49] geometric properties of the drawing including the circularity of the clock circle, placement of clock components, and drawing placement on the page. Clock face A measure of the roundness of the clock face circle. 2.38 2.59 2.36 2.55 circularity [2.21, 2.60] [2.36, 2.83] [2.19, 2.61] [2.33, 2.80] Component A measure of the spatial relationships among the 0.29 0.52 0.32 0.49 placement drawing components. [0.24, 0.37] [0.35, 0.81] [0.27, 0.40] [0.37, 0.69] Vertical spatial A measure of the vertical position of the drawing on 6.75 10.25 10.34 19.77 placement the page. [3.16, 12.12] [4.09, 18.12] [4.58, 17.52] [10.90, 27.98] This analysis results in an easily interpretable hierarchical between the training and testing sets were tested using t-tests for structure: cognitive problems manifest in performance on normally distributed continuous variables, Wilcoxon Rank Sum cognitive features that individual are easily visually understood tests for non-normally distributed variables, and chi-square tests and verified, which translates to lower performance on functional for discrete variables. domain composites associated with those features, and, in turn, To compare DCTclock to existing clock scoring systems, the results in a lower DCTclock score. following scoring systems were operationalized and applied to our clock test data: MoCA (2), Mini-Cog (26), Manos (27), Royall DCTclock Cut Scores (28), Shulman (29), Libon (30), Rouleau (31), and Mendez (32). DCTclock score cutoffs that maximize classification accuracy Full details on the operationalization of these scoring systems between cognitively unimpaired and cognitively impaired have been previously published (10). participants were determined by a Receiver Operating ROC curves were built for scores of DCTclock and the Characteristic (ROC) curve analysis on the development CDT operationalized scoring systems to measure their ability sample. Specifically, the cutoff of 60 for impaired classification to distinguish cognitively unimpaired and cognitively impaired was determined by the Youden Index, an optimal cutoff point participants in the testing set (N = 905). ROC curves were also statistic calculated as sensitivity + specificity–1 (25). The cutoff built for participants in the testing set with both DCTclock scores of 75 for Indeterminate classification was determined via a and MMSE scores within 18 months of each other (N = 591). comparative analysis of possible DCTclock cutoffs to established The area under the ROC curve was calculated for all curves cutoffs used for MMSE classification to facilitate utility in a and differences in AUC between DCTclock and operationalized clinical setting, and was selected to match a cutoff of 28 on the scoring systems and relation between DCTclock and MMSE 0–30 MMSE range. scores were tested using the paired non-parametric test presented by DeLong et al. (33). Sensitivities and specificities between Statistical Analysis and Validation DCTclock scores and MMSE scores were compared at the Procedures optimal cutoff point, determined by the Youden Index (27). All statistical analyses and validation procedures were conducted For each metric developed, differences between cognitively on the testing set, using R version 3.4.2. unimpaired and cognitively impaired groups were tested using Demographic variables were summarized by means and the Wilcoxon Rank Sum test, adjusting for multiple comparisons standard deviations for normally distributed continuous using the Bonferroni-Holm method (34). Additionally, a variables, by median and interquartile range (IQR) for subgroup analysis was conducted using primary consensus non-normally distributed variables, and by frequencies and diagnosis for subjects diagnosed with aMCI and AD and percentages for discrete variables. Differences in demographics compared to cognitively unimpaired. Overall group differences Frontiers in Digital Health | www.frontiersin.org 6 October 2021 | Volume 3 | Article 750661 Souillard-Mandar et al. DCTclock: AI Analysis of Drawing Behavior TABLE 3 | Correlation of DCTclock scores to standard neuropsychological test and 0.90, respectively, while the MMSE had a lower sensitivity scores. (0.67) and comparable specificity (0.90) at the optimal cutoff of Neuropsychological test (n) Correlation, r 2 (95% CI) 27.5 (i.e., <28 vs. ≥28). Figure 3 displays the performance, in terms of AUC, of RBANS (256) 0.59 (0.51, 0.67) DCTclock for different population subsets based on minimum MoCA (85) 0.55 (0.39, 0.69) MMSE score (i.e., subset of subjects who scored > 19, 20, 21, etc. MMSE (772) 0.60 (0.55, 0.64) on the MMSE). DCTclock’s AUC is >0.80, independently of the Trails A (400) 0.66 (0.60, 0.71) subset being analyzed. Trails B (357) 0.60 (0.53, 0.66) Using the primary diagnoses provided by LH, probable AD, Hooper (340) −0.51 (−0.58, −0.43) aMCI, and the FHS cognitively unimpaired subgroups were Beck depression inventory (386) −0.14 (−0.23, −0.04) summarized by MMSE score and DCTclock score. MMSE scores differed significantly (p < 0.001) between the groups, with median [Q1, Q3] of 29 [29, 30], 27 [25.25, 29], and 21 [19, 23] for the cognitively unimpaired, aMCI, and probable AD for both DCTclock and MMSE were tested using the Kruskal- groups, respectively, and some did show the previously reported Wallis test; post-hoc analyses were conducted using the Wilcoxon ceiling effect [40, 41]. DCTclock score also differed significantly Rank Sum test, adjusting for multiple comparisons using the (p < 0.001) between cognitively unimpaired (86.23 [71.79, Bonferroni-Holm method. 93.68]), aMCI (48.50 [38.80, 68.40]), and probable AD (15.45 Construct validity of DCTclock was determined by [6.36, 39.94]). Wilcoxon Rank Sum tests showed significant correlational analyses. For a given NP test, the Pearson differences between cognitively unimpaired and aMCI, as well correlation coefficient was calculated for individuals in the as between aMCI and AD, individually, after adjusting for testing set who took a DCTclock test and the specific NP test multiple comparisons. within 10 days of each other. Thus, the sample size (Table 3) for Within the LH subjects, DCTclock Score correlates well with each NP test varied, as not all participants were administered all the RBANS, MoCA, MMSE, Trails A, Trails B, and Hooper NP tests within that window. tests. This indicates that, in this sample, there is significant agreement between the DCTclock algorithm and these standard RESULTS Neuropsychological tests. Conversely, DCTclock Score does not show a significant correlation with the Beck Depression Demographics Inventory and the Geriatric Depression Scale; the low (close Demographics for both the training and testing data sets are to zero) correlations indicate that DCTclock is not measuring summarized in Table 3. There were no significant differences in indices associated with depression or emotional state, but rather demographic variables between the training and testing groups. are specific to cognition. The mean age for both sets was 69.9, the majority were non- Hispanic, white, college graduates. For both sets, the median MMSE score was 28. Approximately 43% were cognitively DISCUSSION impaired in each dataset. We report on the successful development of sensitive, impactful DCTclock Metrics cognitive tests leveraging the combination of well-known The list of developed metrics and their definitions can be found clinical science with state-of-the-art AI techniques. We in Table 2, along with the median and IQR of each metric demonstrate that it is possible to create an easy to use and across both the cognitively unimpaired and cognitively impaired patient-friendly test that provides sensitive and clinician-usable groups. Every metric was shown to be significantly different information by combining precise behavior capture on large across the two groups at p < 0.01. well-defined population samples, a first layer of AI algorithms for understanding behavior (i.e., mapping pen strokes to drawing Validation of DCTclock symbols), and a second layer that uses more interpretable DCTclock overall score accuracy measures were compared with machine learning models in conjunction with clinician insights the operationalizations of other clock scoring systems using the to create transparent and easily-understood metrics. This overall testing set (N = 921) (Figure 2A). Results indicate that approach, illustrated here with the DCTclock, can be applied to DCTclock score has a significantly higher AUC compared to the other cognitive tasks and other cognitive domains, and promises operationalized scoring systems. To compare the discriminatory to be particularly impactful when multi-modal signals are ability of DCTclock to that of MMSE, ROC analysis was integrated to provide a more sensitive marker of brain function. conducted on the subset of the testing set who were administered Our approach enables the precise analysis of cognitive both DCTclock and the MMSE (N = 591). Construction of a behavioral characteristics that are difficult for clinicians to see and ROC curve for DCTclock scores indicate an AUC of 0.89, while impossible to measure accurately without advanced technologies. the MMSE showed a statistically significantly lower AUC of The CDT has been in clinical use for more than 50 years and has 0.85 (p = 0.029) (Figure 2B). The sensitivity and specificity of diverse scoring systems. Yet, comparison of DCTclock metrics DCTclock at the optimal cutoff of 60 (Youden Index) was 0.74 (Table 2) to eight of the most commonly used traditional scoring Frontiers in Digital Health | www.frontiersin.org 7 October 2021 | Volume 3 | Article 750661 Souillard-Mandar et al. DCTclock: AI Analysis of Drawing Behavior FIGURE 2 | ROC curves with accompanying AUC estimates for DCTclock scores and (A) operationalized clock scoring systems and (B) MMSE. system reveals negligible overlap in the elements that are scored. Our approach offers insights into cognition that are simply not possible using non-digital data and either clinician or AI approaches alone. A first reason is that the process of solving the task previously received no formal evaluation, and the time spent not actively drawing was simply not recorded; in our sample, this accounts for between 53 and 62% of the testing time (Table 2, Percent Think Time). Second, a data-driven approach using AI enables the learning of novel measures of cognition from the data, and weighting them so as to optimize detection of cognitive impairment. This is enabled by large sample sizes of well-defined and precisely characterized subjects for both training and testing sets; a key strength of the development of DCTclock was the opportunity to obtain subject data from research projects, such as the Framingham Heart Study that was integral to this work, FIGURE 3 | AUCs of DCTclock and MMSE, subset using minimum MMSE underscoring the importance of data sharing and harmonization score. of large data repositories. Finally, instead of using black-box AI models which would simply produce an overall score from the raw test data, splitting the analysis into two steps with a focus on clinical interpretability of the results through hierarchical 32 s in our cognitively impaired group (Table 2, Total time). construction of scores from clinically meaningful variables can Combined with the very short test instructions, this enables the enable deeper clinical insights and, ideally, assessment of the total test time to expect to be below 2 min. Beyond the workflow source of impairment. efficiency, this could enable increased sensitivity in detection Validation data shows accuracy, as measured by AUC, of subtle cognitive change that could otherwise be missed due sensitivity, and specificity, substantially greater than the standard to increased stress and fatigue that may be inherent in more CDT and MMSE, as well as the ability to distinguish between intensive and time-consuming tests. For test administrators, cognitively unimpaired, aMCI, and probable AD. Strong the testing protocol remains largely unchanged, with use of correlations with standard neuropsychological screening tests, standard test administration procedures but complemented such as the RBANS and MoCA, underscore the ability of with a fully automated and objective scoring system, obviating this methodology to reliably capture cognitive information in subjective assessment. less time. Importantly, the results demonstrate potential for DCTclock From a patient perspective, this methodology allows for a in differential diagnosis. The lack of correlation with the rapid test that is unintimidating and well-tolerated. The median Beck Inventory demonstrates that the DCTclock algorithm is time for test completion is ∼35 and 25 s for command and detecting cognitive issues exclusive of depression. While more copy clocks, respectively, with those times increasing to 44 and validation will be needed, this could potentially lead to improved Frontiers in Digital Health | www.frontiersin.org 8 October 2021 | Volume 3 | Article 750661 Souillard-Mandar et al. DCTclock: AI Analysis of Drawing Behavior specialist referrals, especially from the perspective of primary care Framingham Heart Study and Lahey Hospital and Medical providers hesitant between a referral to neurology or psychiatry. Center. Requests to access the datasets should be directed There are some limitations and potential improvements to https://biolincc.nhlbi.nih.gov/studies/framoffspring/. to our analysis. First, several race and ethnic groups were under-represented in our samples. DCTclock is designed to ETHICS STATEMENT be applicable to diverse populations and refining DCTclock algorithm performance in widely diverse populations is The studies involving human participants were reviewed and important for future studies. Second, although comments from approved by WIRB (IRB Tracking Number 20160721). The early adopters have been positive, additional studies are also patients/participants provided their written informed consent to needed to assess the feasibility of using DCTclock in diverse participate in this study. clinical settings and to evaluate impact on efficiency, efficacy, and patient outcomes. Third, the dataset was collected over AUTHOR CONTRIBUTIONS a decade, limiting our definition of AD, and aMCI groups to an older non-biomarker standard. AD along with other WS-M model development, contributed to statistical analyses, neurocognitive conditions are moving toward a biomarker- and led the writing of the paper. RD contributed to model based definition of disease detection and progression (35). development, provided input in statistical analyses, and helped Current literature suggest that a substantial proportion of with writing and editing. DP contributed to model development, individuals 55 years or older who are not cognitively impaired Lahey Hospital and Medical Center data supervision and as assessed from standard cognitive tests still have evidence of interpretation, provided input in statistical analyses and clinical amyloid and tau deposition on PET or abnormal CSF consistent interpretation, and helped with writing and editing. BS with neurodegenerative disease, including AD. Therefore, a contributed to statistical analyses and helped with writing and portion of our cognitively unimpaired population may be AD editing. RA Framingham Heart Study data supervision and biomarker positive, contaminating our sample and possibly interpretation, provided input in statistical analyses and clinical reducing sensitivity. Conversely, those in the probable AD/aMCI interpretation, and helped with editing. AP-L contributed to groups may not have biomarker positive AD. It is thus important clinical interpretation of the results and helped with editing. for future validation of DCTclock to study the detection of All authors contributed to the article and approved the biomarker positive against biomarker negative samples and submitted version. to focus on the ability to detect subtle cognitive impairment associated with known diagnostic biomarker measures. Future FUNDING plans include biomarkers studies focusing on measuring the association of DCTclock metrics with PET amyloid and tau. Support for this work came in part from: The Robert E. In fact, one such validation study comparing DCTclock to Wise Research and Education Institution, Defense Advanced PET global amyloid, PET global tau, and the A4 Preclinical Research Projects Agency Contract D13AP00008, National Alzheimer Cognitive Composite (PACC) has already obtained Science Foundation Award IIS-1404494, National Institute interesting results (15), showing that DCTclock performance of Neurological Disorders and Stroke Grants R01-NS17950, was strongly associated with both amyloid and tau deposition in K23-NS60660, and R01-NS082386, National Heart, Lung, and both unimpaired and subtly impaired individuals as measured Blood Institute Contract N01-HC25195, National Institutes by the PACC, and also correlated well with the PACC. Thus, we on Aging Grants R01 AG0333040, AG16492, AG08122, and believe that this novel integrative model of clinician, artificial AG054156, National Institute of Neurological Disorders intelligence, and technology provide exciting opportunities for and Stroke NS017950; National Institute on Mental Health scientific advancement in the understanding cognitive change Grant RO1-MH073989, IH/NCATS Clinical and Translational that may occur in early stage neurodegenerative disorders. Science Award to the University of Florida UL1TR000064, NHLBI, Framingham Heart Study, (NHLBI/NIH contract DATA AVAILABILITY STATEMENT #HHSN268201500001I), the Boston University School of Medicine, the University of Florida Movement Disorders The datasets presented in this article are not readily available and Neurorestoration, National Parkinson Foundation, because data access permission must be requested with the and Pfizer. REFERENCES tool for mild cognitive impairment. J Am Geriatrics Soc. (2005) 53:695– 9. doi: 10.1111/j.1532-5415.2005.53221.x 1. Folstein MF, Folstein SE, McHugh PR. “Mini-mental state”: a practical 3. Barnett JH, Blackwell AD, Sahakian BJ, Robbins TW. The paired associates method for grading the cognitive state of patients for the clinician. learning (PAL) test: 30 years of CANTAB translational neuroscience from J Psychiatric Res. (1975) 12:189–98. doi: 10.1016/0022-3956(75)9 laboratory to bedside in dementia research. Transl Neuropsychopharmacol. 0026-6 (2015) 28:449–74. doi: 10.1007/7854_2015_5001 2. Nasreddine ZS, Phillips NA, Bédirian V, Charbonneau S, Whitehead V, 4. Maruff P, Lim YY, Darby D, Ellis KA, Pietrzak RH, Snyder PJ, et al. Clinical Collin I, et al. The Montreal Cognitive Assessment, MoCA: a brief screening utility of the cogstate brief battery in identifying cognitive impairment in Frontiers in Digital Health | www.frontiersin.org 9 October 2021 | Volume 3 | Article 750661 Souillard-Mandar et al. DCTclock: AI Analysis of Drawing Behavior mild cognitive impairment and Alzheimer’s disease. BMC Psychol. (2013) 25. Youden WJ. Index for rating diagnostic tests. Cancer. (1950) 3:32–5. 1:30. doi: 10.1186/2050-7283-1-30 doi: 10.1002/1097-0142(1950)3:1<32::AIDCNCR2820030106>3.0.CO;2-3 5. Buckley R, Pascual-Leone A. Age-related cognitive decline is indicative of 26. Borson S, Scanlan J, Brush M, Vitaliano P, Dokmak A. The mini- neuropathology. Ann Neurol. (2020) 87:813–15. doi: 10.1002/ana.25733 cog: a cognitive ‘vital signs’ measure for dementia screening in 6. Shulman KI. Clock-drawing: is it the ideal cognitive multi-lingual elderly. Int J Geriatric Psychiatry. (2000) 15:1021– screening test? Int J Geriatric Psychiatry. (2000) 15:548–61. 7. doi: 10.1002/1099-1166(200011)15:11<1021::AID-GPS234>3.0.CO;2-6 doi: 10.1002/1099-1166(200006)15:6<548::AID-GPS242>3.0.CO;2-U 27. Manos PJ, Wu R. The ten point clock test: a quick screen and grading method 7. Shulman KI, Shedletsky R, Silver IL. The challenge of time: clock-drawing for cognitive impairment in medical and surgical patients. Int J Psychiatry and cognitive function in the elderly. Int J Geriatric Psychiatry. (1986) 1:135– Med. (1994) 24:229–44. doi: 10.2190/5A0F-936P-VG8N-0F5R 40. doi: 10.1002/gps.930010209 28. Royall DR, Cordes JA, Polk M. CLOX: an executive clock drawing task. 8. Freedman M, Leach L, Kaplan E, Winocur G, Shulman K, Delis DC. Clock J Neurol Neurosurg Psychiatry. (1998) 64:588–94. doi: 10.1136/jnnp.64. Drawing: A Neuropsychological Analysis. New York, NY: Oxford University 5.588 Press (1994). 29. Shulman KI, Pushkar Gold D, Cohen CA, Zucchero CA. Clock-drawing and 9. Price CC, Cunningham H, Coronado N, Freedland A, Cosentino S, dementia in the community: a longitudinal study. Int J Geriatric Psychiatry. Penney DL, et al. Clock drawing in the Montreal Cognitive Assessment: (1993) 8:487–96. doi: 10.1002/gps.930080606 recommendations for dementia assessment. Dementia Geriatric Cogn Disord. 30. Libon DJ, Swenson RA, Barnoski EJ, Sands LP. Clock drawing as an (2011) 31:179–87. doi: 10.1159/000324639 assessment tool for dementia. Archiv Clin Neuropsychol. (1993) 8:405– 10. Souillard-Mandar W, Davis R, Rudin C, Au R, Libon DJ, Swenson R, 15. doi: 10.1093/arclin/8.5.405 et al. Learning classification models of cognitive conditions from subtle 31. Rouleau I, Salmon DP, Butters N, Kennedy C, McGuire K. Quantitative and behaviors in the digital clock drawing test. Machine Learn. (2016) 102:393– qualitative analyses of clock drawings in Alzheimer’s and Huntington’s 441. doi: 10.1007/s10994-015-5529-5 disease. Brain Cogn. (1992) 18:70–87. doi: 10.1016/0278-2626(92)9 11. Hizel LP, Warner ED, Wiggins ME, Tanner JJ, Parvataneni H, 0112-Y Davis R, et al. Clock drawing performance slows for older adults 32. Mendez MF, Ala T, Underwood KL. Development of scoring criteria for after total knee replacement surgery. Anesthesia Analgesia. (2019) the clock drawing task in Alzheimer’s disease. J Am Geriatrics Soc. (1992) 129:212. doi: 10.1213/ANE.0000000000003735 40:1095–9. doi: 10.1111/j.1532-5415.1992.tb01796.x 12. Piers RJ, Devlin KN, Ning B, Liu Y, Wasserman B, Massaro JM, et al. 33. DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas Age and graphomotor decision making assessed with the digital clock under two or more correlated receiver operating characteristic curves: drawing test: the Framingham Heart Study. J Alzheimer’s Dis. (2017) 60:1611– a nonparametric approach. Biometrics. (1988) 44:837–45. doi: 10.2307/25 20. doi: 10.3233/JAD-170444 31595 13. Libon DJ, Penney DL, Davis R, Tabby DS, Eppig J, Nieves C, et al. Deficits 34. Holm S. A simple sequentially rejective multiple test procedure. Scand J Statist. in processing speed and decision making in relapsing-remitting multiple (1979) 6:65–70. sclerosis: The Digit Clock Drawing Test (dCDT). J Mult Scler. (2014) 35. Jack Jr CR, Bennett DA, Blennow K, Carrillo MC, Dunn B, 1:2. doi: 10.4172/jmso.1000113 Haeberlein SB, et al. NIA-AA research framework: toward a biological 14. Lamar M, Ajilore O, Leow A, Charlton R, Cohen J, GadElkarim J, definition of Alzheimer’s disease. Alzheimer’s Dementia. (2018) et al. Cognitive and connectome properties detectable through individual 14:535–62. doi: 10.1016/j.jalz.2018.02.018 differences in graphomotor organization. Neuropsychologia. (2016) 85:301– 9. doi: 10.1016/j.neuropsychologia.2016.03.034 Conflict of Interest: WS-M was founding VP of AI and Analytics at Digital 15. Rentz DM, Papp KV, Mayblyum DV, Sanchez JS, Klein H, Souillard- Cognition Technologies Inc. (acquired by Linus Health Inc.), VP of AI and Mandar W, et al. Association of digital clock drawing with PET amyloid Analytics at Linus Health Inc., and is now an advisor at Linus Health Inc. DP and tau pathology in normal older adults. Neurology. (2021) 96:e1844– and RD were founders of Digital Cognition Technologies Inc. BS was a Principal 54. doi: 10.1212/WNL.0000000000011697 Statistician at Digital Cognition Technologies Inc. AP-L was a co-founder of Linus 16. Randolph C, Tierney MC, Mohr E, Chase TN. The Repeatable Health and TI Solutions AG; serves on the scientific advisory boards for Starlab Battery for the Assessment of Neuropsychological Status (RBANS): Neuroscience, Magstim Inc., and MedRhythms; and is listed as an inventor on preliminary clinical validity. J Clin Exp Neuropsychol. (1998) several issued and pending patents on the real-time integration of non-invasive 20:310–9. doi: 10.1076/jcen.20.3.310.823 brain stimulation with electroencephalography and magnetic resonance imaging. 17. Bowie CR, Harvey PD. Administration and interpretation of the Trail Making AP-L was partly supported by the National Institutes of Health (R24AG06142, and Test. Nature protocols. (2006) 1:2277–81. doi: 10.1038/nprot.2006.390 P01 AG031720), the National Science Foundation, and the Barcelona Brain Health 18. Boyd JL. A validity study of the Hooper Visual Organization Test. J Consult Initiative primarily funded by La Caixa. RA was scientific advisor to Signant Health Clin Psychol. (1981) 49:15. doi: 10.1037/0022-006X.49.1.15 and a scientific consultant to Biogen. 19. Kaplan E, Goodglass H, Weintraub S. Boston Naming Test. Philadelphia, PA: Lea & Febiger (1983). Publisher’s Note: All claims expressed in this article are solely those of the authors 20. Beck AT, Steer RA, Brown GK. Manual for the Beck Depression Inventory-II. and do not necessarily represent those of their affiliated organizations, or those of San Antonio, TX: Psychol Corp. (1996) doi: 10.1037/t00742-000 the publisher, the editors and the reviewers. Any product that may be evaluated in 21. Satizabal CL, Beiser AS, Chouraki V, Chêne G, Dufouil C, Seshadri S. this article, or claim that may be made by its manufacturer, is not guaranteed or Incidence of dementia over three decades in the Framingham Heart Study. N Engl J Med. (2016) 374:523–32. doi: 10.1056/NEJMoa1504327 endorsed by the publisher. 22. Seshadri S, Beiser A, Kelly-Hayes M, Kase CS, Au R, Kannel WB, et al. The lifetime risk of stroke: estimates from the Framingham Study. Stroke. (2006) Copyright © 2021 Souillard-Mandar, Penney, Schaible, Pascual-Leone, Au and 37:345–50. doi: 10.1161/01.STR.0000199613.38911.b2 Davis. This is an open-access article distributed under the terms of the Creative 23. Krizhevsky A, Sutskever I, Hinton GE. Imagenet classification with deep Commons Attribution License (CC BY). The use, distribution or reproduction in convolutional neural networks. Adv Neural Inform Process Syst. (2012) other forums is permitted, provided the original author(s) and the copyright owner(s) 25:1097–105. doi: 10.1145/3065386 are credited and that the original publication in this journal is cited, in accordance 24. LeCun Y, Cortes C. MNIST Handwritten Digit Database. New Jersey: AT&T with accepted academic practice. No use, distribution or reproduction is permitted Labs (2010). which does not comply with these terms. Frontiers in Digital Health | www.frontiersin.org 10 October 2021 | Volume 3 | Article 750661