Researchers at the University of Berkeley have developed an AI prediction system that harnesses Generative Pretrained Transformer (GPT-4) to forecast future events as accurately as the collective prediction of a group of humans, also known as the ‘wisdom of the crowd’. The AI system built upon the model uses a method called retrieval-augmented reasoning, which involves a multi-step process of information collect, relevancy assessment, summarization, and reasoned forecast with the help of generated search queries and scratchpad prompts.
Further, the researchers used self-supervised fine-tuning to get the system to mimic the best reasoning patterns by fine-tuning the model on past forecasting examples where it outperformed human forecasters. Test results showed that the AI system was particularly effective at questions with high human uncertainty early in the prediction process and when it has access to relevant articles on the subject. Its performance scored higher earlier in the information gathering process.
The AI system is heralded as the first automated forecasting system that performs almost as well as the collective human crowd. It ran into a slight problem when it had too many articles to analyze, possibly because it ‘hedges’ its predictions due to its safety training. Nevertheless, the researchers believe that this form of AI-based prediction could be of great use to policymakers, businesses, and health officials. Making accurate predictions based on language-driven AI forecasting could aid in anticipating and avoiding potential future risks.
While the development of AI for predictive tasks raises ethical questions about transparency, bias, and ethical grounding, AI is already being employed for predictive tasks like policing and welfare decision-making in several democratic countries.
The Berkeley research delineates the potential for AI in making effective forecasts, but it is not clear how the system precisely arrives at its decisions. Therefore, while the application of AI in predicting major societal and individual events might seem somewhat dystopian, it is arguably already a common practice in various parts of the globe.
