Large Language Models (LLMs) often exhibit judgment and decision-making patterns that resemble those of humans, posing them as attractive candidates for studying human cognition. They not only emulate rational norms such as risk and loss aversion, but also showcase human-like errors and biases, particularly in probability judgments and arithmetic operations. Despite their potential prospects, challenges persist in employing LLMs as cognitive models due to their extensive training data and the unclear origins of their human-like behaviors.
Certain concerns revolve around LLMs being trained off significantly larger datasets in comparison to humans, which could artificially improve their human-like behaviors. However, through fine-tuning processes, researchers have managed to enhance LLMs like the LLaMA-1-65B model, improving their predictive accuracy of human behavior. Synthetic datasets have also proven beneficial in refining LLMs’ problem-solving faculties like arithmetic, resulting in improved forecasts of human decisions.
A joint study from Princeton University and Warwick University suggests improving LLMs’ effectiveness as cognitive models by contrasting them with rational agents on computationally equivalent tasks and examining task distributions for LLMs to display human behavior. The Arithmetic-GPT, an LLM pretrained on a realistically valid arithmetic dataset, foretells human behavior more accurately than traditional cognitive models when applied to decision-making situations such as risky and intertemporal choices.
Researchers addressed the challenges in adopting LLMs as cognitive models by devising a data generation algorithm for creating synthetic datasets and exposing the neuronal activation blueprints crucial for decision-making. Arithmetic-GPT, a smaller LLM with Generative Pretrained Transformer (GPT) constitution, was pretrained on arithmetic assignments – greatly facilitating the modeling process. Then, to assess the model’s performance, human decision-making datasets were revisited and reanalyzed.
The study’s results reveal that Arithmetic-GPT embeddings, having been pretrained on ecologically valid synthetic datasets, exhibit the most accurate predictive capability of human choices in decision-making tasks. Compared to models like LLaMA-3-70bInstruct, Arithmetic-GPT embeddings show a stronger correlation with human data, particularly in intertemporal choice tasks – a finding backed up by extensive validation methods.
Compared to traditional cognitive models and some advanced LLMs, including LLaMA-3-70bInstruct, Arithmetic-GPT pretrained on these synthetic datasets exceeds other models in imitating human decision-making behaviors. This novel approach of adopting synthetic datasets and neural activation patterns successfully overcomes various hurdles faced while trying to model human cognition effectively. The robustness demonstrated by the model in accurately predicting human decisions further attests to the potential of LLMs as cognitive models, yielding beneficial insights for both cognitive science and machine learning research.
