As the advancements in Large Language Models (LLMs) such as ChatGPT, LLaMA, and Mistral continue, there are growing concerns about their vulnerability to harmful queries. This has caused an immediate need to implement robust safeguards. Techniques such as supervised fine-tuning (SFT), reinforcement learning from human feedback (RLHF), and direct preference optimization (DPO) have been useful in enhancing the safety of LLMs, and help them reject harmful queries.
However, recent studies have pointed out that LLMs may still be susceptible to complex attack prompts, due to the shortcomings in previous approaches such as DPO, which are limited in only suppressing the activations of toxic parameters without effectively addressing the underlying vulnerabilities. This highlights the necessity to develop precise detoxification methods.
Recently, progress has been made in knowledge editing methods for LLMs, allowing for post-training modifications without any significant impact on the models’ overall performance. Even though this seems like a straightforward solution to detoxifying LLMs, the existing datasets and evaluation metrics have been largely focused on specific harmful issues, ignoring the broader threat of attack prompts and their generalizability to various harmful inputs.
To fill this gap, researchers at Zhejiang University have introduced SafeEdit, a comprehensive benchmark designed to assess detoxification tasks via knowledge editing. SafeEdit encompasses nine unsafe categories with powerful attack templates, extending evaluation metrics to defense success, defense generalization, and overall performance. This offers a standardized structure for evaluating detoxification methods.
Two specific knowledge editing methods, MEND and Ext-Sub, have been used on LLaMA and Mistral models to illustrate the potential to detoxify LLMs efficiently without severely affecting general performance. However, these methods mainly target factual knowledge and may struggle in identifying toxic areas in response to complicated harmful inputs that span multiple sentences.
To combat these issues, a new knowledge editing baseline called Detoxifying with Intraoperative Neural Monitoring (DINM) has been proposed. The DINM method aims to eliminate toxic regions within LLMs while minimizing any undesired effects. Running extensive tests on LLaMA and Mistral models have shown that DINM performs significantly better than traditional SFT and DPO techniques in detoxifying LLMs. This new approach reinforces the significance of accurately pinpointing toxic regions in LLMs.
In conclusion, the study accentuates the notable potential of knowledge editing for detoxifying LLMs. The implementation of SafeEdit provides a reliable structure for evaluation, and the efficient DINM method represents an encouraging move towards solving the challenge of detoxifying LLMs. This research could also influence future applications of SFT, DPO, and knowledge editing in bolstering the safety and resilience of large language models.
The research paper and further information are available on Github. Remember to follow for more updates, and join their ML SubReddit community. If you value their work, you can also sign up for their newsletter. Credit for this research goes to the contributing researchers of this project.
