Large Language Models (LLMs), trained on extensive text data, have displayed unprecedented capabilities in various tasks such as marketing, reading comprehension, and medical analysis. These tasks are usually carried out through next-token prediction and fine-tuning. However, the discernment between deep understanding and shallow memorization among these models remains a challenge. It is essential to assess LLMs’ reasoning faculties through tests that evaluate their generalization capacities beyond training data.
Researchers from MIT presented two studies investigating the efficacy and accuracy of LLMs in predicting outcomes. In the first study, twelve LLMs were employed to predict the outcome of 31 binary questions and then compared their results with 925 human forecasters from a three-month forecasting tournament. The results indicated that the LLMs displayed a performance comparable to human forecasters.
The second study focused on enhancing LLM predictions through human cognitive output, aiming at GPT-4 and Claude 2 models. The primary data collection method involved gathering pre- and post-intervention forecasts per question. The researchers observed how LLMs adjusted behavior regarding human prediction estimates by employing longer prompts for guidance.
The first study collected 1007 forecasts from twelve LLMs, noting predictions were predominantly above the 50% mid-point, indicating a bias towards positive outcomes. The second study analyzed 186 primary and updated forecasts from GPT-4 and Claude 2 across 31 questions. Exposure to human crowd forecasts significantly improved model accuracy and narrowed prediction intervals.
Consequently, the studies demonstrate that when LLMs utilize collective intelligence, they can match the performance of human crowd-based methods in probabilistic forecasting. The findings suggest that combining simpler models can overcome underperformance issues experienced by LLMs. This approach offers practical benefits for numerous real-world applications, potentially equipping decision-makers with accurate political, economic, and technological forecasts, opening the way for wider societal use of LLM predictions.
The research ensures that LLMs continue to be a viable tool in various tasks, including forecasting. While some challenges regarding their understanding and generalization capabilities exist, further enhancements and combined approaches with human output can address these issues and improve their application. The study’s conclusion advocates for the broader implementation of LLMs in making well-informed and calibrated decisions in different societal domains.
