The concept of cascades in large language models (LLMs) has gained popularity for its high task efficiency while reducing data inference. However, potential privacy issues can arise in managing sensitive user information due to interactivity between local and remote models. Conventional cascade systems lack privacy-protecting mechanisms, causing sensitive data to be unintentionally transferred to the remote model or integrated into training sets. This could compromise user privacy and hinder the use of machine learning models in sensitive fields.
However, a research team from Google Research has devised an innovative method that utilizes privacy-preserving techniques in cascade systems. Integrating the social learning paradigm to foster secure query exchanges between the local and remote models. They have utilized data minimization and anonymization techniques along with in-context learning (ICL) abilities of LLMs to form a privacy-conscious connection between local and remote models.
The essence of this technique is to reveal enough relevant information for the remote model to be useful while preserving privacy details. The utilization of gradient-free learning through natural language enables the local model to explain the difficulty to the remote model without sharing the data, thus preserving privacy and benefiting from the remote model’s capabilities.
The research team also explored data from a multitude of datasets to exhibit the capabilities of their approach. Noteworthy improvements were seen in task outcomes when using privacy-preserving cascades compared to non-cascade benchmark scores. For instance, an experiment comprising the local model generating new, unlabeled examples, further labeled by the remote model, attested to a task success rate of 55.9% for math problem-solving and 94.6% for intent recognition, indicating the efficiency of the proposed method while maintaining a low privacy risk.
The study also executed quantitative assessments to gauge the effectiveness of applied privacy-preserving techniques, introducing two metrics: entity leak and mapping leak metrics. The most impressionable privacy conservation was noted when placeholders replaced entities in original examples, with the entity leak metric notably lower compared to other methods.
In summary, this innovative approach opens gateway to use cascade systems integrated with social learning norms and privacy-preserving methods for machine learning without jeopardizing sensitive data. The empirical findings suggest a reduction in privacy risks and boost in task performance, marking the potency of this approach in revolutionizing LLMs’ role in privacy-sensitive applications. The research also conveys the importance of entity leak and mapping leak metrics in determining the effectiveness of privacy-preserving techniques. The research findings credit Google AI on addressing privacy issues in large language models progressively, potentially transforming the role of machine learning in sensitive applications.
