A growing reliance on AI-generated data has led to concerns about model collapse, a phenomenon where a model’s performance significantly deteriorates when trained on synthesized data. This issue has the potential to obstruct the development of methods for efficiently creating high-quality text summaries from large volumes of data.
Currently, the methods used to prevent model collapse often include Reinforcement Learning with Human Feedback (RLHF), data curation, and prompt engineering. However, these techniques often prove to be costly, time-consuming, and not fully reliable. RLHF, for example, while able to improve model performance by incorporating high-quality human-approved data, is not scalable and heavily dependent on human annotators.
Data curation and filtering, on the other hand, can mitigate the impact of low-quality synthesized data, but this also demands considerable effort to maintain training dataset quality. It won’t completely eradicate the risk of model collapse unless robust filtering criteria are used. Similarly, while prompt engineering helps generate higher-quality outputs by guiding the model via specific prompts, it is not foolproof and does require expert knowledge and iterative tests to achieve optimal results.
To overcome these challenges, researchers from Meta AI, NYU, and Peking University have proposed a novel method that applies feedback on synthesized data to prevent model collapse. In contrast to RLHF, this could potentially be partially or fully automated, making it more efficient and scalable.
The emphasis of this proposed methodology is on improving synthesized data via feedback, garnered either from humans or other models. The researchers outlined a theoretical framework demonstrating that a Gaussian mixture classification model could yield optimal performance when trained on this feedback-augmented synthesized data.
The theory was put to the test with two practical experiments. The first focused on training transformers to compute matrix eigenvalues, a task that typically experiences model collapse when relying on purely synthesized data. However, by pruning incorrect predictions and selecting superior guesses from the synthesized data, the model’s performance significantly improved, indicating the effectiveness of reinforcement through data selection.
The second experiment involved news summarization using large language models (LLMs), where feedback-augmented data prevented performance degradation even when the volume of synthesized data increased. This supports the proposition that reinforcement is key to maintaining model integrity.
The researchers also employed a decoding strategy to generate summaries and assessed their performance using the Rouge-1 metric. They utilized a strong verifier model, Llama-3, to select the best-synthesized data for training. Even when only 12.5% of the data was used, the results were superior to the original model trained on the full dataset, suggesting that the proposed method effectively counteracts model collapse.
By incorporating feedback mechanisms to refine the quality of synthetic data, these findings show promise for avoiding model collapse in LLMs trained on synthesized data. Apart from ensuring sustained model performance, this approach also offers a scalable and cost-effective alternative to traditional RLHF methods, potentially paving the way for more robust and reliable AI systems in the future.
