6f66993c0d55252878399e8e3d4182a4 pnas_a2023v120n51art2309583120.pdf a942318bedd6f40df5c3e80f1332c2b379b2d32a pnas_a2023v120n51art2309583120.pdf d20bc852686f3b23d36e74ed315801ecf91a3ef32a9abed0308709765a89a9da pnas_a2023v120n51art2309583120.pdf Title: Systematic testing of three Language Models reveals low language accuracy, absence of response stability, and a yes-response bias Subject: Proc. Natl. Acad. Sci. U.S.A. 2023.120:e2309583120 Author: Creator: Adobe InDesign 15.1 (Windows) Producer: Adobe PDF Library 15.0; modified using iText 4.2.0 by 1T3XT CreationDate: Mon Dec 11 19:17:00 2023 CET ModDate: Tue Feb 6 20:52:50 2024 CET Tagged: no UserProperties: no Suspects: no Form: none JavaScript: no Pages: 10 Encrypted: no Page size: 584.784 x 783 pts Page rot: 0 File size: 422871 bytes Optimized: no PDF version: 1.4 name type encoding emb sub uni object ID ------------------------------------ ----------------- ---------------- --- --- --- --------- ECYRYK+Montserrat-Bold Type 1C Custom yes yes yes 39 0 AMWOUS+STIX-Bold Type 1C WinAnsi yes yes yes 237 0 KGWWEY+STIX-Regular Type 1C Custom yes yes yes 40 0 NKLTUS+OpenSans-Light CID TrueType Identity-H yes yes yes 43 0 ISAUCC+OpenSans CID TrueType Identity-H yes yes yes 41 0 AOXTUS+OpenSans-Bold CID TrueType Identity-H yes yes yes 71 0 QMZLKM+OpenSans TrueType WinAnsi yes yes yes 60 0 DQLLKM+OpenSans-Light TrueType WinAnsi yes yes yes 238 0 RYMRYK+OpenSans-Bold TrueType WinAnsi yes yes yes 42 0 NXNYUS+OpenSans-Semibold CID TrueType Identity-H yes yes yes 61 0 ZLBYGU+AGaramondPro-Regular Type 1C Custom yes yes yes 38 0 SAVNMI+AGaramondPro-Semibold Type 1C Custom yes yes yes 239 0 QGNMKM+AGaramondPro-Italic Type 1C WinAnsi yes yes yes 62 0 Times-Roman Type 1 WinAnsi no no no 2 0 LRWDSW+AvenirNextLTPro-CnIt Type 1C Custom yes yes yes 44 0 DKZMKM+AvenirNextLTPro-Cn Type 1C Custom yes yes yes 45 0 UXDAOE+AvenirNextLTPro-BoldCn Type 1C WinAnsi yes yes yes 46 0 PSQWWO+OpenSans-BoldItalic CID TrueType Identity-H yes yes yes 58 0 HBUAOE+OpenSans-BoldItalic TrueType WinAnsi yes yes yes 59 0 BSSQIQ+OpenSans-Italic TrueType WinAnsi yes yes yes 70 0 BSSQIQ+Helvetica Type 1C WinAnsi yes yes no 72 0 BSSQIQ+Helvetica Type 1C WinAnsi yes yes no 79 0 BSSQIQ+Helvetica Type 1C WinAnsi yes yes no 84 0 BSSQIQ+Helvetica Type 1C WinAnsi yes yes no 85 0 BSSQIQ+Helvetica Type 1C WinAnsi yes yes no 86 0 BSSQIQ+Helvetica Type 1C WinAnsi yes yes no 93 0 BSSQIQ+Helvetica Type 1C WinAnsi yes yes no 94 0 BSSQIQ+Helvetica Type 1C WinAnsi yes yes no 95 0 UFGAOE+AvenirNextLTPro-DemiCn Type 1C WinAnsi yes yes yes 102 0 Jhove (Rel. 1.28.0, 2023-05-18) Date: 2024-02-07 02:30:37 CET RepresentationInformation: pnas_a2023v120n51art2309583120.pdf ReportingModule: PDF-hul, Rel. 1.12.4 (2023-03-16) LastModified: 2024-02-06 20:58:27 CET Size: 422871 Format: PDF Version: 1.4 Status: Well-Formed and valid SignatureMatches: PDF-hul MIMEtype: application/pdf PDFMetadata: Objects: 0 FreeObjects: 2 IncrementalUpdates: 0 DocumentCatalog: ViewerPreferences: HideToolbar: false HideMenubar: false HideWindowUI: false FitWindow: false CenterWindow: false DisplayDocTitle: false NonFullScreenPageMode: UseNone Direction: L2R ViewArea: CropBox ViewClip: CropBox PrintArea: CropBox PageClip: CropBox PageLayout: SinglePage PageMode: UseNone Language: en-US Outlines: Item: Title: Systematic testing of three Language Models reveals low language accuracy, absence of response stability, and a yes-response bias Children: Item: Title: Significance Item: Title: Grammaticality Judgments Item: Title: Materials and Methods Children: Item: Title: Sentence Material. Item: Title: LMs. Item: Title: Human Data. Item: Title: Results Children: Item: Title: Accuracy. Children: Item: Title: Accuracy by model and condition. Children: Item: Title: Differences between grammatical phenomena. Item: Title: Stability. Children: Item: Title: Stability by model and condition. Item: Title: The Interplay between Stability and Accuracy. Children: Item: Title: Does the lack of accuracy follow from the lack of stability? Item: Title: Does accuracy improve over repetitions? Item: Title: Does stability improve over repetitions? Item: Title: Comparisons with Human Data. Children: Item: Title: Accuracy. Children: Item: Title: Stability. Item: Title: Does accuracy improve over repetitions? Item: Title: Does stability improve over repetitions? Item: Title: Discussion Item: Title: Data, Materials, and Software Availability Item: Title: ACKNOWLEDGMENTS Item: Title: Supporting Information Children: Item: Title: Anchor 35 Filters: FilterPipeline: FlateDecode Fonts: Type0: Font: BaseFont: AOXTUS+OpenSans-Bold Encoding: Identity-H ToUnicode: true Font: BaseFont: ISAUCC+OpenSans Encoding: Identity-H ToUnicode: true Font: BaseFont: PSQWWO+OpenSans-BoldItalic Encoding: Identity-H ToUnicode: true Font: BaseFont: NKLTUS+OpenSans-Light Encoding: Identity-H ToUnicode: true Font: BaseFont: NXNYUS+OpenSans-Semibold Encoding: Identity-H ToUnicode: true Type1: Font: BaseFont: ZLBYGU+AGaramondPro-Regular FontSubset: true FirstChar: 25 LastChar: 151 FontDescriptor: FontName: ZLBYGU+AGaramondPro-Regular Flags: Serif, Nonsymbolic FontBBox: -183, -354, 1355, 917 FontFile3: true EncodingDictionary: BaseEncoding: WinAnsiEncoding Differences: true ToUnicode: true Font: BaseFont: UFGAOE+AvenirNextLTPro-DemiCn FontSubset: true FirstChar: 48 LastChar: 57 FontDescriptor: FontName: UFGAOE+AvenirNextLTPro-DemiCn Flags: Nonsymbolic FontBBox: -121, -244, 912, 959 FontFile3: true Encoding: WinAnsiEncoding ToUnicode: true Font: BaseFont: ECYRYK+Montserrat-Bold FontSubset: true FirstChar: 31 LastChar: 121 FontDescriptor: FontName: ECYRYK+Montserrat-Bold Flags: Nonsymbolic FontBBox: -882, -266, 1679, 1076 FontFile3: true EncodingDictionary: BaseEncoding: WinAnsiEncoding Differences: true ToUnicode: true Font: BaseFont: KGWWEY+STIX-Regular FontSubset: true FirstChar: 30 LastChar: 45 FontDescriptor: FontName: KGWWEY+STIX-Regular Flags: Serif, Nonsymbolic FontBBox: -970, -486, 2208, 1023 FontFile3: true EncodingDictionary: BaseEncoding: WinAnsiEncoding Differences: true ToUnicode: true Font: BaseFont: BSSQIQ+Helvetica FontSubset: true FirstChar: 32 LastChar: 121 FontDescriptor: FontName: BSSQIQ+Helvetica Flags: Nonsymbolic FontBBox: -951, -481, 1446, 1122 FontFile3: true Encoding: WinAnsiEncoding Font: BaseFont: LRWDSW+AvenirNextLTPro-CnIt FontSubset: true FirstChar: 30 LastChar: 248 FontDescriptor: FontName: LRWDSW+AvenirNextLTPro-CnIt Flags: Nonsymbolic, Italic FontBBox: -189, -244, 829, 951 FontFile3: true EncodingDictionary: BaseEncoding: WinAnsiEncoding Differences: true ToUnicode: true Font: BaseFont: AMWOUS+STIX-Bold FontSubset: true FirstChar: 32 LastChar: 45 FontDescriptor: FontName: AMWOUS+STIX-Bold Flags: Serif, Nonsymbolic FontBBox: -998, -552, 2000, 1055 FontFile3: true Encoding: WinAnsiEncoding ToUnicode: true Font: BaseFont: DKZMKM+AvenirNextLTPro-Cn FontSubset: true FirstChar: 29 LastChar: 252 FontDescriptor: FontName: DKZMKM+AvenirNextLTPro-Cn Flags: Nonsymbolic FontBBox: -105, -244, 846, 953 FontFile3: true EncodingDictionary: BaseEncoding: WinAnsiEncoding Differences: true ToUnicode: true Font: BaseFont: UXDAOE+AvenirNextLTPro-BoldCn FontSubset: true FirstChar: 110 LastChar: 119 FontDescriptor: FontName: UXDAOE+AvenirNextLTPro-BoldCn Flags: Nonsymbolic FontBBox: -120, -244, 939, 964 FontFile3: true Encoding: WinAnsiEncoding ToUnicode: true Font: BaseFont: SAVNMI+AGaramondPro-Semibold FontSubset: true FirstChar: 30 LastChar: 146 FontDescriptor: FontName: SAVNMI+AGaramondPro-Semibold Flags: Serif, Nonsymbolic FontBBox: -177, -282, 1123, 893 FontFile3: true EncodingDictionary: BaseEncoding: WinAnsiEncoding Differences: true ToUnicode: true Font: BaseFont: BSSQIQ+Helvetica FontSubset: true FirstChar: 32 LastChar: 118 FontDescriptor: FontName: BSSQIQ+Helvetica Flags: Nonsymbolic FontBBox: -951, -481, 1446, 1122 FontFile3: true Encoding: WinAnsiEncoding Font: BaseFont: BSSQIQ+Helvetica FontSubset: true FirstChar: 32 LastChar: 121 FontDescriptor: FontName: BSSQIQ+Helvetica Flags: Nonsymbolic FontBBox: -951, -481, 1446, 1122 FontFile3: true Encoding: WinAnsiEncoding Font: BaseFont: BSSQIQ+Helvetica FontSubset: true FirstChar: 32 LastChar: 121 FontDescriptor: FontName: BSSQIQ+Helvetica Flags: Nonsymbolic FontBBox: -951, -481, 1446, 1122 FontFile3: true Encoding: WinAnsiEncoding Font: BaseFont: BSSQIQ+Helvetica FontSubset: true FirstChar: 46 LastChar: 118 FontDescriptor: FontName: BSSQIQ+Helvetica Flags: Nonsymbolic FontBBox: -951, -481, 1446, 1122 FontFile3: true Encoding: WinAnsiEncoding Font: BaseFont: BSSQIQ+Helvetica FontSubset: true FirstChar: 32 LastChar: 121 FontDescriptor: FontName: BSSQIQ+Helvetica Flags: Nonsymbolic FontBBox: -951, -481, 1446, 1122 FontFile3: true Encoding: WinAnsiEncoding Font: BaseFont: QGNMKM+AGaramondPro-Italic FontSubset: true FirstChar: 49 LastChar: 115 FontDescriptor: FontName: QGNMKM+AGaramondPro-Italic Flags: Serif, Nonsymbolic, Italic FontBBox: -182, -338, 1668, 917 FontFile3: true Encoding: WinAnsiEncoding ToUnicode: true Font: BaseFont: BSSQIQ+Helvetica FontSubset: true FirstChar: 32 LastChar: 121 FontDescriptor: FontName: BSSQIQ+Helvetica Flags: Nonsymbolic FontBBox: -951, -481, 1446, 1122 FontFile3: true Encoding: WinAnsiEncoding Font: BaseFont: BSSQIQ+Helvetica FontSubset: true FirstChar: 46 LastChar: 121 FontDescriptor: FontName: BSSQIQ+Helvetica Flags: Nonsymbolic FontBBox: -951, -481, 1446, 1122 FontFile3: true Encoding: WinAnsiEncoding TrueType: Font: BaseFont: BSSQIQ+OpenSans-Italic FontSubset: true FirstChar: 49 LastChar: 68 FontDescriptor: FontName: BSSQIQ+OpenSans-Italic Flags: Nonsymbolic, Italic FontBBox: -498, -268, 1231, 1048 FontFile2: true Encoding: WinAnsiEncoding ToUnicode: true Font: BaseFont: RYMRYK+OpenSans-Bold FontSubset: true FirstChar: 32 LastChar: 124 FontDescriptor: FontName: RYMRYK+OpenSans-Bold Flags: Nonsymbolic FontBBox: -619, -293, 1319, 1069 FontFile2: true Encoding: WinAnsiEncoding ToUnicode: true Font: BaseFont: HBUAOE+OpenSans-BoldItalic FontSubset: true FirstChar: 32 LastChar: 121 FontDescriptor: FontName: HBUAOE+OpenSans-BoldItalic Flags: Nonsymbolic, Italic FontBBox: -513, -293, 1293, 1069 FontFile2: true Encoding: WinAnsiEncoding ToUnicode: true Font: BaseFont: QMZLKM+OpenSans FontSubset: true FirstChar: 42 LastChar: 100 FontDescriptor: FontName: QMZLKM+OpenSans Flags: Nonsymbolic FontBBox: -550, -271, 1204, 1048 FontFile2: true Encoding: WinAnsiEncoding ToUnicode: true Font: BaseFont: DQLLKM+OpenSans-Light FontSubset: true FirstChar: 44 LastChar: 99 FontDescriptor: FontName: DQLLKM+OpenSans-Light Flags: Nonsymbolic FontBBox: -521, -269, 1140, 1048 FontFile2: true Encoding: WinAnsiEncoding ToUnicode: true CIDFontType2: Font: BaseFont: ISAUCC+OpenSans CIDSystemInfo: Registry: Adobe Registry: Identity Supplement: 0 FontDescriptor: FontName: ISAUCC+OpenSans Flags: Symbolic FontBBox: -550, -271, 1204, 1048 FontFile2: true Font: BaseFont: AOXTUS+OpenSans-Bold CIDSystemInfo: Registry: Adobe Registry: Identity Supplement: 0 FontDescriptor: FontName: AOXTUS+OpenSans-Bold Flags: Symbolic FontBBox: -619, -293, 1319, 1069 FontFile2: true Font: BaseFont: NXNYUS+OpenSans-Semibold CIDSystemInfo: Registry: Adobe Registry: Identity Supplement: 0 FontDescriptor: FontName: NXNYUS+OpenSans-Semibold Flags: Symbolic FontBBox: -584, -282, 1261, 1049 FontFile2: true Font: BaseFont: NKLTUS+OpenSans-Light CIDSystemInfo: Registry: Adobe Registry: Identity Supplement: 0 FontDescriptor: FontName: NKLTUS+OpenSans-Light Flags: Symbolic FontBBox: -521, -269, 1140, 1048 FontFile2: true Font: BaseFont: PSQWWO+OpenSans-BoldItalic CIDSystemInfo: Registry: Adobe Registry: Identity Supplement: 0 FontDescriptor: FontName: PSQWWO+OpenSans-BoldItalic Flags: Symbolic, Italic FontBBox: -513, -293, 1293, 1069 FontFile2: true XMP: application/pdf Systematic testing of three Language Models reveals low language accuracy, absence of response stability, and a yes-­response bias 2023-12-11T23:47+05:30 2024-02-06T11:52:50-08:00 2024-02-06T11:52:50-08:00 Adobe InDesign 15.1 (Windows) uuid:240e7f06-1504-46b0-bc09-23d60d0a00e1 xmp.did:FC7F117407206811822A8C10FEE461B6 xmp.id:44710dcc-6051-764e-8e99-852d16af3569 proof:pdf xmp.iid:b4ec4e40-4783-0c45-a7b8-57c64fe41012 xmp.did:3072cbbc-9e2f-654f-8b41-4504a4ff2135 xmp.did:FC7F117407206811822A8C10FEE461B6 default converted from application/x-indesign to application/pdf Adobe InDesign 15.1 (Windows) / 2023-12-11T23:47+05:30 Adobe PDF Library 15.0; modified using iText 4.2.0 by 1T3XT False 3759392515 MyriadPro-Light Adobe Systems 1.006 0 Myriad Pro 3759392515 OpenType - PS 0 2771909978 MyriadPro-Bold Adobe Systems 1.006 0 Myriad Pro 2771909978 OpenType - PS 0 172538106 MyriadPro-LightSemiCn Adobe Systems 1.003 0 Myriad Pro 172538106 OpenType - PS 0 2496651361 MyriadPro-Regular Adobe Systems 1.006 0 Myriad Pro 2496651361 OpenType - PS 0 3198020658 MyriadPro-Semibold Adobe Systems 1.006 0 Myriad Pro 3198020658 OpenType - PS 0 3024036851 MyriadPro-SemiCn Adobe Systems 1.003 0 Myriad Pro 3024036851 OpenType - PS 0 3349345699 MyriadPro-SemiboldSemiCn Adobe Systems 1.003 0 Myriad Pro 3349345699 OpenType - PS 0 4267955085 MyriadPro-SemiboldSemiCnIt Adobe Systems 1.003 0 Myriad Pro 4267955085 OpenType - PS 0 2876027303 MyriadPro-BoldSemiCnIt Adobe Systems 1.003 0 Myriad Pro 2876027303 OpenType - PS 0 1229766490 MyriadPro-BoldSemiCn Adobe Systems 1.003 0 Myriad Pro 1229766490 OpenType - PS 0 Pages: Page: Sequence: 1 Annotations: Annotation: Subtype: Link Rect: 525, 116, 548, 108 Annotation: Subtype: Link Rect: 391, 108, 506, 100 Annotation: Subtype: Link Rect: 391, 90, 460, 82 Annotation: Subtype: Link Rect: 391, 71, 548, 63 Annotation: Subtype: Link Rect: 391, 64, 479, 56 Annotation: Subtype: Link Rect: 96, 657, 96, 646 Annotation: Subtype: Link Rect: 95, 659, 107, 646 Annotation: Subtype: Link Rect: 155, 659, 168, 646 Annotation: Subtype: Link Rect: 241, 659, 253, 646 Annotation: Subtype: Link Rect: 517, 763, 550, 729 Page: Sequence: 2 Annotations: Annotation: Subtype: Link Rect: 327, 614, 393, 603 Annotation: Subtype: Link Rect: 374, 583, 433, 572 Page: Sequence: 3 Annotations: Annotation: Subtype: Link Rect: 58, 549, 118, 538 Annotation: Subtype: Link Rect: 36, 465, 111, 452 Page: Sequence: 4 Page: Sequence: 5 Page: Sequence: 6 Page: Sequence: 7 Page: Sequence: 8 Page: Sequence: 9 Annotations: Annotation: Subtype: Link Rect: 405, 246, 465, 236 Annotation: Subtype: Link Rect: 49, 113, 158, 105 Annotation: Subtype: Link Rect: 503, 113, 529, 105 Annotation: Subtype: Link Rect: 314, 105, 392, 97 Page: Sequence: 10 Annotations: Annotation: Subtype: Link Rect: 212, 588, 282, 580 Annotation: Subtype: Link Rect: 133, 556, 237, 548 Annotation: Subtype: Link Rect: 212, 364, 260, 356 Annotation: Subtype: Link Rect: 49, 356, 89, 348 Annotation: Subtype: Link Rect: 356, 740, 460, 732 Annotation: Subtype: Link Rect: 314, 636, 404, 628 Annotation: Subtype: Link Rect: 411, 620, 518, 612 Annotation: Subtype: Link Rect: 314, 540, 418, 532 Annotation: Subtype: Link Rect: 423, 501, 531, 490 Annotation: Subtype: Link Rect: 395, 476, 499, 468 Annotation: Subtype: Link Rect: 331, 436, 435, 428 Annotation: Subtype: Link Rect: 422, 413, 534, 402 Annotation: Subtype: Link Rect: 314, 405, 405, 394 Annotation: Subtype: Link Rect: 501, 365, 546, 354 Checksum: 21033ce8 Type: CRC32 Checksum: 6f66993c0d55252878399e8e3d4182a4 Type: MD5 Checksum: a942318bedd6f40df5c3e80f1332c2b379b2d32a Type: SHA-1 Checksum: d20bc852686f3b23d36e74ed315801ecf91a3ef32a9abed0308709765a89a9da Type: SHA-256