Evaluating the Deductive Competence of Large Language Models

Authors

Spencer M. Seals, Wright State University
Valerie L. Shalin, Wright State University - Main CampusFollow

Document Type

Article

Publication Date

1-1-2024

Identifier/URL

41341196 (Pure)

Abstract

The development of highly fluent large language models (LLMs) has prompted increased interest in assessing their reasoning and problem-solving capabilities. We investigate whether several LLMs can solve a classic type of deductive reasoning problem from the cognitive science literature. The tested LLMs have limited abilities to solve these problems in their conventional form. We performed follow up experiments to investigate if changes to the presentation format and content improve model performance. We do find performance differences between conditions; however, they do not improve overall performance. Moreover, we find that performance interacts with presentation format and content in unexpected ways that differ from human performance. Overall, our results suggest that LLMs have unique reasoning biases that are only partially predicted from human reasoning performance and the human-generated language corpora that informs them.

Comments

Publisher Copyright: ©2024 Association for Computational Linguistics.

Repository Citation

Seals, S. M., & Shalin, V. L. (2024). Evaluating the Deductive Competence of Large Language Models. Long Papers, 8606-8622.
https://corescholar.libraries.wright.edu/psychology/644

DOI

10.48550/arXiv.2309.05452