Introduction
Generative artificial intelligence is developing at a rapid pace today. It is used to create chatbots, search assistants, tools for programming, design, medicine, and dozens of other areas. At the heart of this progress are large models trained on huge amounts of data.
This data is usually collected from open sources on the internet, aggregated, cleaned, and filtered. During the dataset preparation stage, various heuristics, rules, and automatic filters are applied to remove toxic, low-quality, or obviously problematic content. However, even the most advanced data cleaning pipelines cannot guarantee perfect results.
In practice, this leads to various risks:
1. Leaks of confidential information: the model may reproduce personal data or fragments of sensitive documents.
2. Copyright infringement: training corpora contains a huge amount of text, images, and code that are protected by law.
3. Hallucinations and biases: models tend to invent facts, confidently answer questions without sufficient grounds, and transmit social or cultural stereotypes inherited from the data.
Approaches to data protection and their limitations
Ofcourse, AI Security is currently undergoing rapid development, researching methods for protecting data and models from undesirable behavior and information leaks. In general, the main protection methods can be divided into the following broad categories:
1. System or user prompts: allow you to set rules for model behavior at the user interaction level.
2. Classifiers: BERT-type models or other ML architectures that can classify undesirable content and filter it before it reaches the user.
3. Statistical rules: these include regular expressions, blacklists of words and patterns, which are mainly used in the post-processing stage.
4. SFT/RL alignment: involves retraining the model on “correct” behavior through SFT or RLHF. This approach allows the desired behavior to be embedded in the model and increases control over generation.
5. RAG: uses a database of examples of good and bad behavior. The model receives few-shot prompts through vector search, which allows its generation to be quickly directed.
Limitations of basic protection methods
Although there are many methods for protecting models — from prompts and classifiers to SFT/RL alignment and RAG — all of these approaches have their limitations:
1. New attacks bypass protection: New types of attacks are emerging that allow existing security measures to be bypassed. As a result, there is no such thing as a completely secure model today.
2. Scaling issues between models: protection methods do not scale well between different model families. What works effectively on one model, such as the Llama family, does not guarantee the same result on another, such as the Qwen family.
3. Data remains in the model: personal data and “bad” content still remain inside the model. Protection methods are mainly aimed at preventing the model from generating unwanted content, but they do not remove this data from the model itself.
Legal requirements
In addition to technical limitations, working with data in large models is increasingly subject to legal requirements.
One of the key documents here is the GDPR (General Data Protection Regulation) — the European Union’s general data protection regulation. It establishes strict rules for the collection, processing, and storage of personal data of individuals within the EU. One of the central provisions of the GDPR is the so-called right to be forgotten, which allows any person to request that a company delete their personal data if it has been collected or used without legal grounds. In case of refusal or inability to fulfill such a request, the company may be subject to serious fines.
Thus, legal requirements directly push developers to manage knowledge within models, rather than just controlling their behavior at the level of interfaces and filters. This raises two key questions:
1. How to remove “bad” or unwanted data from the model?
2. How to maintain or even improve the performance of the model while doing so?
Is retraining not an option?
Ofcourse, we can take our model and retrain it from scratch, excluding unwanted or problematic data from the training dataset, but in practice, this approach faces a number of serious limitations:
1. Retraining takes significantly more time: in essence, we are going through the entire training process again, which at least doubles the development time.
2. The cost increases: training large models means repeated expenses for computing resources, primarily GPUs, which translates into direct financial costs.
3. Risk of quality deterioration: even after retraining, there is no guarantee that the new version of the model will maintain the same level of quality. Performance may decline, and the model’s behavior may change in unpredictable ways.
4. Requests to delete data may come repeatedly: first from one user, then from several, and so on. Retraining the model from scratch after each such request is labor-intensive, expensive, and does not scale well in real-world conditions.
We would like to circumvent these limitations while complying with legal requirements. This is where Machine Unlearning comes into play.
Machine Unlearning — what is it?
Press enter or click to view image in full size
In general, machine unlearning is a field concerned with developing methods that allow models to “forget” or exclude data on which they were trained without the need for complete retraining of the model.
What can we forget?
Machine Unlearning can be used to remove various types of information from a model:
– Individual elements — specific confidential user data that must be removed for legal or ethical reasons.
– Attributes — data grouped by common characteristics. For example, in a resume screening system, you can remove features such as the gender or race of candidates to minimize bias when processing thousands of applications.
– Class removal — all information about specific objects or people. For example, in a facial recognition system, you can remove data about specific people so that the model no longer recognizes their faces.
– Deleting individual tasks — deleting knowledge about a specific task. Some tasks may become irrelevant over time, and deleting them frees up memory in the model for retraining on new, higher-priority tasks without compromising its performance.
Data fundamentals for model unlearning
Asin machine learning, data plays a key role in Machine Unlearning. While in conventional ML, data is divided into train (for model training) and test (for testing its performance on new data), machine unlearning uses a slightly different division structure.
Training data is divided into two sets:
– Forget Set — data that the model must forget.
– Retain Set — data that the model must retain in order to maintain its previous performance. As a rule, this is all data except for the Forget Set.
The test dataset also consists of two parts:
– Unseen Forget Set — data from the Forget Set that the model did not see during the training phase and that it should forget.
– Unseen Retain Set — data from the Retain Set that is used to verify that the model retains its capabilities after removing unwanted knowledge.
Forget Set Construction
Data for the Forget Set is generated in one or more ways:
– User definition: the user specifies in advance the specific data that needs to be deleted.
– Search in the training corpus: clustering or nearest neighbor search methods can be used to identify similar examples that may be subject to deletion.
– Model hacking: in cases where data cannot be directly located in the training corpus, various attacks on models are used to identify them. For example, Membership inference attacks allow you to determine whether a model was trained on a specific example, and Model inversion attacks allow you to recover approximate source data based on model parameters or its responses.
Constructing the Retain Set
Data for the Retain Set is formed in similar ways:
– User definition: the user specifies in advance the data that the model should retain and for which quality must be maintained after the unlearning process.
– The remainder of the original dataset: after the Forget Set is defined, all other data from the original training corpus automatically forms the Retain Set.
– External or alternative datasets: Retain Set may include other datasets that the model may not have been trained on previously, but which are used to verify that the unlearning process has not led to performance degradation on irrelevant Forget Set tasks.
The main stages of model unlearning
Before considering the machine unlearning pipeline, it is useful to recall the standard machine learning process. It usually consists of the following stages:
1. Data collection — we form a dataset for training the model.
2. Data preprocessing — we clean the data, create new features, and prepare them for training.
3. Model training — we train the model on the prepared data with the selected hyperparameters.
4. Quality assessment — we test the model on a test dataset. If the results satisfy the quality metrics, the model is sent to production and its behavior is monitored; if not, we return to one of the previous stages and retrain it.
The machine unlearning pipeline is largely similar to classic ML, but adds new steps for knowledge removal. The main feature is that a request to delete certain data comes from outside. This process includes:
1. Identifying data for deletion: we form the Forget Set and Retain Set based on requests and data analysis.
2. Model unlearning: the main process in which the model “forgets” information from the Forget Set, while the Retain Set is used to preserve its performance and correct behavior.
3. Testing and evaluation: we test the new “unlearned” model on the Unseen Forget Set and Unseen Retain Set to ensure that the data has indeed been deleted and that performance on other tasks has been preserved.
Effective machine unlearning: where and how to apply it
In addition to the main scenarios for applying machine unlearning — compliance with legal requirements, deletion of confidential data, and reduction of bias — machine unlearning can also be considered as an alternative to classic approaches to model alignment, such as RLHF.
Machine Unlearning as an alternative to RLHF
Machine Unlearning as an alternative to RLHF
The method itself is based on teaching the model “correct” behavior using human feedback and usually includes the following steps:
1. Supervised Fine-Tuning (SFT): the model is retrained on a dataset labeled by humans to reinforce the basic patterns of desired behavior.
2. Collecting human preferences: annotators evaluate the model’s responses and form preference pairs or ratings.
3. Training the reward model: a separate model learns to predict human preferences based on input queries and responses.
4. Re-training the main model: the main model is optimized using reinforcement learning to maximize the expected reward from the reward model.
However, RLHF has a number of significant limitations:
High data costs: collecting and labeling high-quality human feedback requires significant time and financial resources.
Resource-intensive training: the process involves training multiple models and a large number of iterations, making it expensive in terms of computational resources.
Subjectivity of feedback: human evaluations inevitably reflect limited or biased preferences, which can lead to ambiguous model behavior and hidden biases.
Here, machine unlearning can serve as an effective alternative to RLHF. Instead of aligning with user preferences and training multiple models, we can selectively remove unwanted knowledge from an already trained model. This allows us to significantly reduce computational costs, time, and resources, since we essentially only need the Forget Set, rather than the entire corpus of labeled data and complex iterative RL cycles.
Open source tool for unlearning: OpenUnlearning
Overall, the community around Machine Unlearning is still forming, but the first tools for practical work are already appearing. One such tool is OpenUnlearning, which provides implementations of various unlearning methods, as well as sets of metrics and benchmarks for evaluating the effectiveness of forgetting information.
However, this tool has its limitations: in its current implementation, the project relies on fine-tuning with updating all model parameters, which makes it very demanding on memory and GPU computing resources.
To reduce resource requirements, we forked the project and integrated support for LoRA (Low-Rank Adaptation), a parameter-efficient retraining method in which the basic model weights are frozen and training occurs only for a small number of additional parameters, which allows for an order of magnitude reduction in GPU memory consumption while maintaining comparable quality.
Conclusions
Machine unlearning is a new field that allows selective deletion or “forgetting” of unwanted knowledge, protecting personal data, reducing bias, and maintaining model performance.
This article was the first and introductory part of a series on Machine Unlearning. In the following parts, we will take a closer look at methods for evaluating the effectiveness of unlearning and maintaining performance, different approaches to model unlearning itself, benchmarks, and much more!
Useful resources
2. Article about model inversion attacks
3. Open Source tool OpenUnlearning
