Testing AI on language comprehension tasks reveals insensitivity to underlying meaning
Dentella, Vittoria (Universitat Rovira i Virgili)
Günther, Fritz (Humboldt-Universität zu Berlin)
Murphy, Elliot (University of Texas Health Science Center at Houston (UTHealth))
Marcus, Gary (New York University)
Leivada, Evelina (Universitat Autònoma de Barcelona. Departament de Filologia Catalana)
Institució Catalana de Recerca i Estudis Avançats

Date:	2024
Description:	11 pàg.
Abstract:	Large Language Models (LLMs) are recruited in applications that span from clinical assistance and legal support to question answering and education. Their success in specialized tasks has led to the claim that they possess human-like linguistic capabilities related to compositional understanding and reasoning. Yet, reverse-engineering is bound by Moravec's Paradox, according to which easy skills are hard. We systematically assess 7 state-of-the-art models on a novel benchmark. Models answered a series of comprehension questions, each prompted multiple times in two settings, permitting one-word or open-length replies. Each question targets a short text featuring high-frequency linguistic constructions. To establish a baseline for achieving human-like performance, we tested 400 humans on the same prompts. Based on a dataset of n = 26,680 datapoints, we discovered that LLMs perform at chance accuracy and waver considerably in their answers. Quantitatively, the tested models are outperformed by humans, and qualitatively their answers showcase distinctly non-human errors in language understanding. We interpret this evidence as suggesting that, despite their usefulness in various tasks, current AI models fall short of understanding language in a way that matches humans, and we argue that this may be due to their lack of a compositional operator for regulating grammatical and semantic information.
Rights:	Aquest document està subjecte a una llicència d'ús Creative Commons. Es permet la reproducció total o parcial, la distribució, i la comunicació pública de l'obra, sempre que no sigui amb finalitats comercials, i sempre que es reconegui l'autoria de l'obra original. No es permet la creació d'obres derivades.
Language:	Anglès
Document:	Article ; recerca ; Versió publicada
Subject:	Humans ; Comprehension/physiology ; Language ; Artificial Intelligence ; Linguistics ; Semantics ; Female
Published in:	Scientific reports, Vol. 14 Núm. 1 (2024) , p. 28083, ISSN 2045-2322

DOI: 10.1038/s41598-024-79531-8
PMID: 39543236

11 p, 2.9 MB

The record appears in these collections:
Articles > Research articles
Articles > Published articles