Under Review (R&R - Preprint Available)
Trent N. Cash, Daniel M. Oppenheimer, Sara Christie, Mira Devgan
View PDF
Cite
Cite
APA
Click to copy
Cash, T. N., Oppenheimer, D. M., Christie, S., & Devgan, M. Quantifying Uncert-AI-nty: Testing the Accuracy of LLMs’ Confidence Judgments . https://doi.org/10.31234/osf.io/47df5_v3
Chicago/Turabian
Click to copy
Cash, Trent N., Daniel M. Oppenheimer, Sara Christie, and Mira Devgan. “Quantifying Uncert-AI-Nty: Testing the Accuracy of LLMs’ Confidence Judgments ”(n.d.).
MLA
Click to copy
Cash, Trent N., et al. Quantifying Uncert-AI-Nty: Testing the Accuracy of LLMs’ Confidence Judgments . doi:10.31234/osf.io/47df5_v3.
BibTeX Click to copy
@article{trent-a,
title = {Quantifying Uncert-AI-nty: Testing the Accuracy of LLMs’ Confidence Judgments },
doi = {10.31234/osf.io/47df5_v3},
author = {Cash, Trent N. and Oppenheimer, Daniel M. and Christie, Sara and Devgan, Mira}
}
Abstract
The rise of Large Language Model (LLM) chatbots, such as ChatGPT and Gemini, has revolutionized how we access information. These LLMs can answer a wide array of questions, including those with inherent uncertainties, such as predictions about future events. The present studies investigate the capability of LLMs to quantify such uncertainty through confidence judgments. We evaluate the absolute and relative accuracy of confidence judgments from two LLMs (ChatGPT and Gemini) compared to human participants across three prediction domains: NFL game winners (Study 1a; n = 502), Oscar award winners (Study 1b; n = 109), and future Pictionary performance (Study 2; n = 164). Our findings reveal that LLMs’ confidence judgments closely align with those of humans in terms of accuracy, biases, and errors. However, unlike humans, LLMs struggle to adjust their confidence judgments based on past performance, highlighting a key area for improvement in their design.