Alzheimer’s & Dementia: Translational Research & Clinical Interventions 5 (2019) 933-938 Short Report A metabolite-based machine learning approach to diagnose Alzheimer-type dementia in blood: Results from the European Medical Information Framework for Alzheimer disease biomarker discovery cohort Daniel Stamatea,b,c, Min Kimd, Petroula Proitsie, Sarah Westwoodf, Alison Bairdf, Alejo Nevado- Holgadof, Abdul Hyee, Isabelle Bosg,h, Stephanie J. B. Vosg, Rik Vandenbergheh, Charlotte E. Teunisseni, Mara Ten Kateh,i, Philip Scheltensh, Silvy Gabelj,k,l, Karen Meersmansk,l, Olivier Blinm, Jill Richardsonn, Ellen De Roecko,p,q, Sebastiaan Engelborghsp,q,r, Kristel Sleegersq,s, Regis Bordett, Lorena Ramitu, Petronella Kettunenv, Magda Tsolakiw, Frans Verheyg, Daniel Alcoleax, Alberto Lleox, Gwendoline Peyratouty, Mikel Taintaz, Peter Johannsenaa, Yvonne Freund-Levie,bb, Lutz Fr€ olichcc, Valerija Dobricicdd, Giovanni B. Frisoniee,ff, Jose L. Molinuevot,gg, Anders Wallinhh, Julius Poppy,ii, Pablo Martinez- Lagez, Lars Bertramdd,jj, Kaj Blennowkk,ll, Henrik Zetterbergkk,ll,mm,nn, Johannes Strefferoo, Pieter J. Visserg,h, Simon Lovestonef,pp, Cristina Legido-Quigleyd,qq,* a Division of Population Health, Health Services Research and Primary Care, University of Manchester, Manchester, UK b Data Science & Soft Computing Lab, London, UK c Computing Department, Goldsmiths College, University of London, London, UK d Steno Diabetes Center Copenhagen, Gentofte, Denmark e Institute of Psychiatry, Psychology and Neuroscience, Maurice Wohl Clinical Neuroscience Institute, King’s College London, London, UK f Department of Psychiatry, University of Oxford, Oxford, UK g Department of Psychiatry and Neuropsychology, School for Mental Health and Neuroscience, Alzheimer Centrum Limburg, Maastricht University, Maastricht, the Netherlands h Department of Neurology, Alzheimer Center Amsterdam, Amsterdam Neuroscience, Vrije Universiteit Amsterdam, Amsterdam UMC, Amsterdam, the Netherlands i Department of Radiology and Nuclear Medicine, VU University Medical Center, Amsterdam, the Netherlands j Department of Clinical Chemistry, Neurochemistry Laboratory, Amsterdam Neuroscience, Amsterdam University Medical Centers, Vrije Universiteit, the Netherlands k University Hospital Leuven, Leuven, Belgium l Department of Neurosciences, Laboratory for Cognitive Neurology, KU Leuven, Belgium m AIX Marseille University, INS, Ap-hm, Marseille, France n Neurosciences Therapeutic Area, GlaxoSmithKline R&D, Stevenage, UK o Faculty of Psychology & Educational Sciences Vrije Universiteit Brussel (VUB), Brussels, Belgium p Reference Center for Biological Markers of Dementia (BIODEM), University of Antwerp, Antwerp, Belgium q Institute Born-Bunge, University of Antwerp, Antwerp, Belgium r Department of Neurology, UZ Brussel and Center for Neurosciences, Vrije Universiteit Brussel (VUB), Brussels, Belgium s Neurodegenerative Brain Diseases Group, Center for Molecular Neurology, VIB, Belgium t University of Lille, Inserm, CHU Lille, Lille, France u Alzheimer’s Disease & Other Cognitive Disorders Unit, Hospital Clınic-IDIBAPS, Barcelona, Spain v Institute of Neuroscience and Physiology, Sahlgrenska Academy at University of Gothenburg, Gothenburg, Sweden w 1st Department of Neurology, AHEPA University Hospital, Makedonia, Thessaloniki, Greece *Corresponding author. Tel.: +4530913083. E-mail address: cristina.legido.quigley@regionh.dk https://doi.org/10.1016/j.trci.2019.11.001 2352-8737/ Ó 2019 The Authors. Published by Elsevier Inc. on behalf of the Alzheimer’s Association. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/). 934 D. Stamate et al. / Alzheimer’s & Dementia: Translational Research & Clinical Interventions 5 (2019) 933-938 x Memory Unit, Neurology Department, Hospital de la Santa Creu i Sant Pau, Barcelona, Spain y University Hospital of Lausanne, Lausanne, Switzerland z Center for Research and Advanced Therapies, Fundacion CITA-alzheimer Fundazioa, Donostia/San Sebastian, Spain aa Danish Dementia Research Centre, Rigshospitalet, Copenhagen University Hospital, Copenhagen, Denmark bb Department of Neurobiology, Caring Sciences and Society (NVS), Division of Clinical Geriatrics, Karolinska Institute, and Department of Geriatric Medicine, Karolinska University Hospital Huddinge, Stockholm, Sweden cc Department of Geriatric Psychiatry, Zentralinstitut f€ur Seelische Gesundheit, University of Heidelberg, Mannheim, Germany dd L€ubeck Interdisciplinary Platform for Genome Analytics, Institutes of Neurogenetics and Cardiogenetics, University of L€ubeck, L€ubeck, Germany ee University of Geneva, Geneva, Switzerland ff IRCCS Istituto Centro San Giovanni di Dio Fatebenefratelli, Brescia, Italy gg Barcelona Beta Brain Research Center, Unversitat Pompeu Fabra, Barcelona, Spain hh Institute of Neuroscience and Physiology, Sahlgrenska Academy at the University of Gothenburg, Gothenburg, Sweden ii Department of Mental Health and Psychiatry, Geriatric Psychiatry, Geneva University Hospitals, Geneva, Switzerland jj Department of Psychology, University of Oslo, Oslo, Norway kk Department of Psychiatry and Neurochemistry, Institute of Neuroscience and Physiology, University of Gothenburg, M€olndal, Sweden ll Clinical Neurochemistry Laboratory, Sahlgrenska University Hospital, M€olndal, Sweden mm UK Dementia Research Institute at UCL, London, UK nn Department of Neurodegenerative Disease, UCL Institute of Neurology, London, UK oo Reference Center for Biological Markers of Dementia (BIODEM), Institute Born-Bunge, University of Antwerp, Antwerp, Belgium pp Janssen-Cilag UK Ltd, Oxford, UK qq Institute of Pharmaceutical Science, King’s College London, London, UK Abstract Introduction: Machine learning (ML) may harbor the potential to capture the metabolic complexity in Alzheimer Disease (AD). Here we set out to test the performance of metabolites in blood to cate- gorize AD when compared to CSF biomarkers. Methods: This study analyzed samples from 242 cognitively normal (CN) people and 115 with AD-type dementia utilizing plasma metabolites (n 5 883). Deep Learning (DL), Extreme Gradient Boosting (XGBoost) and Random Forest (RF) were used to differentiate AD from CN. These models were internally validated using Nested Cross Validation (NCV). Results: On the test data, DL produced the AUC of 0.85 (0.80–0.89), XGBoost produced 0.88 (0.86– 0.89) and RF produced 0.85 (0.83–0.87). By comparison, CSF measures of amyloid, p-tau and t-tau (together with age and gender) produced with XGBoost the AUC values of 0.78, 0.83 and 0.87, respectively. Discussion: This study showed that plasma metabolites have the potential to match the AUC of well- established AD CSF biomarkers in a relatively small cohort. Further studies in independent cohorts are needed to validate whether this specific panel of blood metabolites can separate AD from controls, and how specific it is for AD as compared with other neurodegenerative disorders. Ó 2019 The Authors. Published by Elsevier Inc. on behalf of the Alzheimer’s Association. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/ 4.0/). Keywords: EMIF-AD; Alzheimer’s disease; Metabolomics; Biomarkers; Machine-Learning 1. Introduction progression [5,6]. Therefore, blood AD markers could potentially aid early diagnosis and recruitment for trials. At present, the diagnosis of Alzheimer disease–type de- Here we utilized data generated as part of the European mentia (AD) is based on protein biomarkers in cerebrospinal Medical Information Framework for AD Multimodal fluid (CSF) and brain imaging together with a battery of Biomarker Discovery (EMIF-AD) previously reported in cognition tests. Diagnostic tools based on CSF collection full in Kim et al. [7]. As discussed in that paper, metabolite are invasive while brain-imaging tools are still costly, levels were measured using liquid chromatography–mass and therefore, there is a need to identify noninvasive spectroscopy (LC-MS) to cover ca. Eight hundred metabo- tools for early detection as well as for measuring disease pro- lites and these metabolites related to CSF biomarkers of gression. AD commonly used in clinical research including trials, In recent years, an increasing number of studies have and increasingly in clinical practice, as part of the diagnostic examined blood metabolites as potential AD biomarkers work up. Here we explore the potential of different Machine [1–4]. The advantages of looking at blood metabolites are Learning (ML) algorithms to identify those individuals with that they are easily accessible but also that they represent AD from dataset and to compare the effectiveness of blood- an essential aspect of the phenotype of an organism and based metabolites as an indicator of clinical diagnosis to that hence might act as a molecular fingerprint of disease of CSF markers. In this study we employed two state-of-the- D. Stamate et al. / Alzheimer’s & Dementia: Translational Research & Clinical Interventions 5 (2019) 933-938 935 art ML algorithms—Deep Learning (DL) and Extreme 3. Results Gradient Boosting (XGBoost)—and compared these to the more commonly utilized Random Forest (RF) algorithm. In this study, we analyzed metabolite data derived from blood samples from 357 participants (CN n 5 242, AD n 5 115) previously reported in Kim et al. [7]. Demographic 2. Methods and clinical data can be found in [7]; in short, there was no This study accessed data previously generated from 242 difference in gender while AD participants were older samples from cognitively normal (CN) individuals and 115 when compared with CN participants. from people with AD-type dementia (AD) samples in On the test data, the DL model produced a Receiver which diagnosis was based on clinical diagnosis. Details Operating Characteristic (ROC) Area Under the Curve on the subjects, clinical and cognitive data, as well as mea- (AUC) value of 0.85 with its 95% confidence interval surements of AD pathological markers have been (CI) ranging between 0.8038 and 0.8895. The XGBoost described elsewhere [7,8]. The metabolomics data em- model produced the AUC value of 0.88 (95% CI [0.8619, ployed here was accessed in the EMIF-AD portal and the 0.8903]). When the classifier model RF was employed, acquisition and processing details can be found via open the resulting AUC was 0.85 (95%CI [0.8323, 0.8659]). access in [7]. In short, the EMIF-AD cohort is a collated Fig. 1 illustrates ROC curves obtained from the three ML cohort making use of existing data and samples collected models. in 11 different studies across Europe, with the aim to The MC simulation conducted with XGBoost, which discover novel diagnostic and prognostic markers for pre- was the superior predictive model in our analysis, led to a dementia AD. Gaussian distribution of the AUC values according to In the current study, the main objective was to use state- [11] and as confirmed by Shapiro-Wilk test (P of-the-art ML classification algorithms to build CN versus. value 5 .6819). The 50 AUC values obtained in MC had AD predictive models using blood metabolites. For this pur- a minimum of 0.8614, a maximum of 0.8923, a mean of pose, we employed DL and XGBoost. Additionally we also 0.8761, a median of 0.8766 and a standard deviation of employed the more popularly used RF algorithm. These 0.0072. The t-test showed that the true mean of AUC for models were compared in terms of binary classifiers with XGBoost applied on plasma metabolites was not lower Area Under the Curve (AUC) in Receiver Operating Charac- than 0.87 (P value 5 1.265 ! 10207). teristic (ROC) curves. For comparison, we also investigated the levels of am- The metabolites with more than 45% missing values yloid, p-tau and t-tau, to which we added also age and were discarded. The remaining missing values were gender, and their prediction for clinical AD versus CN. handled with imputation methods based on the k-nearest XGBoost models were built in the same manner as for neighbor (RF and DL), or internally by the classification al- metabolite predictors. Together with age and gender, gorithm (XGBoost). Models were built and evaluated using a Nested Cross Validation (NCV), which used 9/10 data folds for model training and optimization in an inner cross-validation, and 1/10 data folds for model testing in an outer cross-validation. The process was repeated 10 times, for each of the test data folds. The analysis was further extended by assessing the stabil- ity of the AUC performance with Monte Carlo (MC) simula- tions consisting of 50 repeated similar NCV experiments. As such, multiple models were built on multiple samples in the NCV and MC, using metabolite predictors selected on the basis of their capability to discriminate CN versus AD as measured by the Relief algorithm [9] applied on training data in combination with 500 permutations of the outcome variables’ values. This method computes the predictors’ importance defined as the standardized Relief score, accord- ing to Measuring Predictor Importance chapter of [10]. Part of the prediction modeling methodology in this study was adapted after [11], with different algorithms, and followed recommendations from [10,12]. The analysis was carried out using R software [13]. Pathway analysis was performed on the top 20 ranked metabolites using MetaboAnalyst 4.0 [14]. The algorithms were run on four servers with 6-core Fig. 1. Shows the AUC values for the XGBoost, RF and DL models. Xeon CPUs and 336 GB RAM. XGBoost performed best with metabolite predictors in the EMIF cohort. 936 D. Stamate et al. / Alzheimer’s & Dementia: Translational Research & Clinical Interventions 5 (2019) 933-938 amyloid led to AUC 0.78 (95%CI [0.7626, 0.8013]); p-tau AUC. DL algorithms are known to often take advantage led to AUC 0.83 (95%CI [0.8188, 0.8470]); and t-tau led of large and/or unstructured data (such as images) to pro- to AUC 0.87 (95%CI [0.8583, 0.8854]). From the mean duce more accurate category discrimination/prediction. AUC for metabolites and for amyloid, p-tau and t-tau In a study using the Alzheimer’s Disease Neuroimaging calculated individually, the t-tests showed superior Initiative (ADNI) data for AD prediction, XGBoost values for metabolites (P value,2.2 ! 10216, demonstrated superior results (AUC 5 0.97 (0.01) when P value,2.2 ! 10216 and P value 5 .005921, respec- including imaging parameters (MRI and PET) as predictors tively). and when compared to RF, Support Vector Machines, The top 20 ranked predictors out of the 347 selected by Gaussian Processes and Stochastic Gradient Boosting the method presented in the previous section are shown in [16]. In another study where cognition and MRI were Fig. 2. used as predictors, Kernel Ridge Regression was performed Pathway analyses revealed that the Nitrogen pathway was to R2 5 0.87 (0.025) when cognition and MRI predictors overrepresented (qFDR 5 0.004) within the panel. Molecules were included [17]. that were captured as the 20 top ranking predictors are dis- Pathway analyses using the top 20 AD predicting metab- cussed in the next section. olites derived from the Relief method showed that the nitro- gen pathway was overrepresented. Some of the molecules selected have been reported in metabolomics studies and 4. Discussion have been implicated in neurodegeneration: dodecanoate, Machine Learning applied to healthcare is increasingly which is a C12 fatty acid, was found correlated to longitu- enabled by the advent of high-performance computing and dinal measures of cognition in the ADNI cohort [3] and so the development of complex algorithms. In this study, we was the bile acid glycolithocholate, which was associated employed two state-of-the-art algorithms, DL and to both AD and cognition measures (ADAS-Cog13) in XGBoost, and a more conventional algorithm, RF, to obtain one of the biggest cross-sectional studies on cognition, high accuracy models to predict AD versus CN with metab- AD and the microbiome [18]. Plasmalogens were also olites as predictors. Our study showed that the best model found in decreased levels in our cohort in agreement with was based on XGBoost [15], which is an enhanced form an earlier report [19]. The amide form of vitamin B3, nico- of Gradient Boosting Machines methods based on decision tinamide, has been implicated in both neuroprotection and trees [12]. In our study RF and DL achieved comparable neuronal death [20]. Fig. 2. The x-axis shows the top 20 ranked predictors, and the y-axis shows the predictors’ importance computed as the standardized relief score according to Measuring Predictor Importance chapter of [10]. D. Stamate et al. / Alzheimer’s & Dementia: Translational Research & Clinical Interventions 5 (2019) 933-938 937 New metabolites that could be of interest and have not within the fifth framework program (QLRT-2001-2455). been previously reported as related to AD were phytanate The EDAR study was funded by the European Commission and furoylglycine. The former is a known neurotoxin which within the fifth framework program (contract no. 37670). impairs mitochondrial function and transcription [21]. Furo- The San Sebastian GAP study is partially funded by the ylglycine is a metabolite which, as lithocholic acid, is mainly Department of Health of the Basque Government (allocation synthesized by the microbiome and has been reported as a 17.0.1.08.12.0000.2.454.01. 41142.001.H). Kristel Sleegers biomarker of coffee consumption [22]. is supported by the Research Fund of the University of Ant- A limitation of our study is that it does not include an werp. Daniel Stamate is supported by the Alzheimer’s external validation due to the size of the cohort. However, Research UK (ARUK-PRRF2017-012). we implemented a NCV procedure repeated 50 times in a MC simulation that led to an extended internal validation with prediction accuracy of cases. Further studies will assess RESEARCH IN CONTEXT the performance of ratios/combinations of CSF markers and metabolites, life-style factors and disorders commonly found in the elderly, together with testing the specificity for this specific panel in other neurodegenerative (e.g., PD, 1 Systematic review: The authors reviewed the litera- FTD), neurological (e.g., stroke) and psychiatric (e.g., ture using PubMed and reported key publications. depression) disorders associated with aging. Most AD biomarker studies employing state-of-the- The intent of this paper was to compare the performance art machine learning (ML) techniques utilized neu- of different ML algorithms to identify people with AD from roimaging data. Those which looked at blood me- cognitively unimpaired individuals. Here we show first that tabolomics data were small and used clinical all three approaches used demonstrate good discriminatory diagnosis as an endpoint. Subsequently, we explored power, second that XGBoost is somewhat more effective the potential of state-of-the art ML algorithms in this particular dataset than RF and DL and third, that including Deep Learning and Extreme Gradient this accuracy for clinical diagnosis is broadly similar to Boosting to test the performance of blood metabolite that achieved by CSF markers of AD pathology. The lack levels to clinical diagnosis and compared to CSF of a replication and validation dataset limits the interpreta- biomarkers. tion of this finding, but nonetheless, the strong prediction 2 Interpretation: The results in here show that with of diagnostic category from a blood-based metabolite state-of-the-art ML algorithms, blood metabolites biomarker set is further evidence of the potential of such ap- have the potential to match the CSF markers of AD proaches to complement other biomarkers in identification pathology on identifying people with AD from of people with likely AD. cognitively unimpaired individuals. All the ML al- gorithms employed showed good discriminatory po- Acknowledgments wer. The authors thank the individuals and families who took 3 Future directions: Results of this study should be part in this research. The authors would also like to thank replicated and validated using an independent dataset. all people involved in data and sample collection and/or Further studies will also aim to assess the perfor- logistics across the different centers, and in particular mance of ratios/combinations of CSF markers and Marije Benedictus, Wiesje van de Flier, Charlotte Teunis- metabolites, life-style factors and disorders sen, Ellen De Roeck, Naomi De Roeck, Ellis Niemants- commonly found in the elderly, together with testing verdriet, Charisse Somers, Babette Reijs, Andrea the specificity for this specific panel in other neuro- Izagirre Otaegi, Mirian, Ecay Torres, Sindre Rolstad, degenerative (e.g., PD, FTD), neurological (e.g., Eva Bringman, Domile Tautvydaite, Barbara Moullet, stroke) and psychiatric (e.g., depression) disorders Charlotte Evenepoel, Isabelle Cleynen, Bea Bosch, associated with aging. Daniel Alcolea Rodriguez, Moira Marizzoni, Alberto Re- dolfi and Paolo Bosco. Funding: The present study was conducted as part of the EMIF-AD project, which has received support from the Innovative Medicines Initiative Joint Undertaking under EMIF grant agreement no. 115372, resources of which are References composed of financial contribution from the European Union’s Seventh Framework Program (FP7/2007–2013) [1] Proitsi P, Kim M, Whiley L, Pritchard M, Leung R, Soininen H, et al. and EFPIA companies’ in-kind contribution. The DE- Plasma lipidomics analysis finds long chain cholesteryl esters to be SCRIPA study was funded by the European Commission associated with Alzheimer’s disease. Transl Psychiatry 2015;5:e494. 938 D. Stamate et al. / Alzheimer’s & Dementia: Translational Research & Clinical Interventions 5 (2019) 933-938 [2] Proitsi P, Kim M, Whiley L, Simmons A, Sattlecker M, Velayudhan L, [14] Chong J, Soufan O, Li C, Caraus I, Li S, Bourque G, et al. MetaboA- et al. Association of blood lipids with Alzheimer’s disease: A compre- nalyst 4.0: towards more transparent and integrative metabolomics hensive lipidomics analysis. Alzheimers Dement 2017;13:140–51. analysis. Nucleic Acids Res 2018;46:W486–94. [3] Toledo JB, Arnold M, Kastenmuller G, Chang R, Baillie RA, Han X, [15] Chen T, Guestrin C. XGBoost: A Scalable Tree Boosting System. Pro- et al. Metabolic network failures in Alzheimer’s disease: A biochem- ceedings of the 22nd ACM SIGKDD International Conference on ical road map. Alzheimers Dement 2017;13:965–84. Knowledge Discovery and Data Mining. San Francisco, California, [4] Varma VR, Oommen AM, Varma S, Casanova R, An Y, Andrews RM, USA: ACM; 2016. p. 785–94. et al. Brain and blood metabolite signatures of pathology and progres- [16] Stamate D, Alghamdi W, Ogg J, Hoile R, Murtagh F. A Machine sion in Alzheimer disease: A targeted metabolomics study. Plos Med Learning Framework for Predicting Dementia and Mild Cognitive 2018;15:e1002482. Impairment. 2018 17th IEEE International Conference on Machine [5] Fiehn O. Metabolomics – the link between genotypes and phenotypes. Learning and Applications (ICMLA); 2018. p. 671–8. Plant Mol Biol 2002;48:155–71. [17] Bucholc M, Ding X, Wang H, Glass DH, Wang H, Prasad G, et al. A [6] Whiley L, Legido-Quigley C. Current strategies in the discovery of practical computerized decision support system for predicting the small-molecule biomarkers for Alzheimer’s disease. Bioanalysis severity of Alzheimer’s disease of an individual. Expert Syst Appl 2011;3:1121–42. 2019;130:157–71. [7] Kim M, Snowden S, Suvitaival T, Ali A, Merkler DJ, Ahmad T, et al. [18] MahmoudianDehkordi S, Arnold M, Nho K, Ahmad S, Jia W, Xie G, Primary fatty amides in plasma associated with brain amyloid burden, et al. Altered bile acid profile associates with cognitive impairment in hippocampal volume, and memory in the European Medical Informa- Alzheimer’s disease-An emerging role for gut microbiome. Alz- tion Framework for Alzheimer’s Disease biomarker discovery cohort. heimers Dement 2019;15:76–92. Alzheimer’s Demen 2019;15:817–27. [19] Han X, Holtzman DM, McKeel DW Jr. Plasmalogen deficiency in [8] Bos I, Vos S, Vandenberghe R, Scheltens P, Engelborghs S, Frisoni G, early Alzheimer’s disease subjects and in animal models: molecular et al. The EMIF-AD Multimodal Biomarker Discovery study: design, characterization using electrospray ionization mass spectrometry. methods and cohort characteristics. Alzheimers Res Ther 2018;10:64. J Neurochem 2001;77:1168–80. [9] Kononenko I. Estimating attributes: Analysis and extensions of RELIEF. [20] Fricker RA, Green EL, Jenkins SI, Griffin SM. The Influence of Nico- Berlin, Heidelberg: Springer Berlin Heidelberg; 1994. p. 171–82. tinamide on Health and Disease in the Central Nervous System. Int J [10] Kuhn M, Johnson K. Applied predictive modeling. Springer; 2013. Tryptophan Res 2018;11:1178646918776658. [11] Stamate D, Katrinecz A, Stahl D, Verhagen SJW, Delespaul P, van [21] Schonfeld P, Reiser G. Brain Lipotoxicity of Phytanic Acid and Very Os J, et al. Identifying psychosis spectrum disorder from experience Long-chain Fatty Acids. Harmful Cellular/Mitochondrial Activities in sampling data using machine learning approaches. Schizophr Res Refsum Disease and X-Linked Adrenoleukodystrophy. Aging Dis 2019;209:156–63. 2016;7:136–49. [12] Hastie T, Tibshirani R, Friedman J. The Elements of Statistical [22] Radjursoga M, Karlsson GB, Lindqvist HM, Pedersen A, Persson C, Learning. 2 ed. New York: Springer-Verlag; 2009. Pinto RC, et al. Metabolic profiles from two different breakfast meals [13] Team RC. R: A Language and Environment for Statistical Computing; characterized by (1)H NMR-based metabolomics. Food Chem 2017; 2019. 231:267–74.