Pretrained language models (LMs) are essential tools in the realm of machine learning, often used for a variety of tasks and domains. But, adapting these models, also known as finetuning, can be expensive and time-consuming, especially for larger models. Traditionally, the solution to this issue has been to use Parameter-efficient finetuning (PEFT) methods such as Adapters and LoRA.
However, researchers from Stanford and Pr(Ai)2R Group are now pioneering a new method called Representation Finetuning (ReFT), which proposes a shift from modifying model weights to modifying model representations. Traditional PEFT methods have largely ignored model representations, despite them being rich with semantic information. ReFT methods challenge this approach by training interventions onto a small fraction of model representations, thus steering model behaviors to help solve downstream tasks.
A key instance of this approach is the Low-rank Linear Subspace ReFT (LoReFT), which intervenes on hidden representations within a low-rank projection matrix’s linear subspace. This method builds on existing methods such as distributed alignment search (DAS) and has demonstrated modern performance on various benchmarks. Notably, this is achieved while using significantly fewer parameters than traditional PEFT methods.
ReFT methods serve to advance research into neural network interpretability, challenging traditional interpretation approaches which focus on individual neurons in isolation. The approach also offers more efficient alternatives to weight-based PEFTs, suggesting it may be worthwhile to explore these methods across different model families and domains.
Looking forward, future research directions for ReFT could include exploring its applicability across different model families, automating the hyperparameter search process, and investigating effective interventions for specific tasks. Researchers will also seek to understand the power of learned orthogonal subspaces, giving new insights into model interpretation.
The challenge now is to establish fair and consistent evaluation practices. This includes creating appropriate benchmarks that allow for comparison between PEFTs and ReFTs, ensuring proper hyperparameter-tuning comparisons, and mitigating overfitting to ensure performance assessment aligns with real-world scenarios. The work of these researchers sets the stage for a significant shift in machine learning, challenging traditional approaches and offering new paths for efficiency and effectiveness in the field.
