Large Language Model (LLM) chatbots - such as Chat GPT, Gemini, and Claude - are incredibly powerful artificial intelligence tools that have skyrocketed in popularity over the last few years. LLMs can edit writing, generate code, create images, and answer nearly any question that we might ask. Due to their proprietary nature, the mechanisms underlying LLMs' cognitive (or pseudo-cognitive) processes are largely unknown to users. However, the extent to which LLMs themselves are aware of their cognitive processes is an open question. 

Psychologists refer to knowledge about one's own cognitive processes as metacognition. In humans, metacognition is a powerful tool for learning more quickly, communicating more accurately, making better decisions, and engaging in all-around more effective cognition. If LLMs are capable of metacognition, it could prove to be a powerful mechanism for improving their accuracy, communication skills, and general usability.

In this line of research, I implement experimental paradigms from the metacognition literature to test the metacognitive capacities of LLMs. Some questions that I will explore in this domain include:

Cash, T. N., & Oppenheimer, D. M. (In Press).  Generative chatbots AIn’t experts: Exploring cognitive and metacognitive limitations that hinder expertise in generative chatbots. Journal of Applied Research in Memory & Cognition. 

Cash, T. N., Oppenheimer, D. M., & Christie, S. (R&R). Quantifying UncertAInty: Testing the Accuracy of LLMs’ Confidence Judgments. Preprint: https://doi.org/10.31234/osf.io/47df5 

Cash, T. N., Oppenheimer, D. M., & Christie, S. (2024, November). Can LLMs tell you when they might be wrong? Evaluating the accuracy of LLMs’ confidence judgments [Poster Session]. Annual Meeting of the Society for Judgment and Decision Making, New York, NY.

Cash, T. N., & Oppenheimer, D. M. (2024, June). Evaluating the accuracy of LLMs' confidence judgments: A study of NFL predictions [Oral Presentation]. International Conference on Thinking, Milan, Italy.