The Setup
In March 2023, Italy’s data protection authority, the Garante, did something no other Western regulator had done: it ordered OpenAI to stop processing Italian users’ data and temporarily blocked access to ChatGPT in the country. The core complaint was that OpenAI had no clear legal basis for collecting personal data to train its models, and no meaningful mechanism to let users correct or delete information the model had absorbed.
OpenAI restored access within a month after publishing a privacy policy update and adding an opt-out form. The Garante declared partial victory. But the underlying question, the one that made lawyers and engineers quietly uncomfortable, never got a clean answer: if your data trained the model, can deleting your account actually undo that?
The honest answer is no. And understanding why matters for anyone who uses these tools, builds them, or advises people who do.
What Happened
When you use a product like ChatGPT, Claude, or Gemini, your conversations may (depending on your settings and the company’s policies) be used to improve the model. This doesn’t mean your exact messages are stored in a database somewhere, retrievable on demand. It means your interactions influence the weights of a neural network during a training run.
Think of it like this: you tell a friend something confidential, and it subtly changes how they think about a topic going forward. You can ask them to forget what you said. They might genuinely try. But the belief it formed is still there, woven into how they reason.
Neural network weights are the same. They’re the product of billions of parameter updates applied across enormous datasets. No specific piece of training data corresponds to a single retrievable memory. The model doesn’t “remember” your conversation the way a database row does. It has been shaped by it.
This is what makes GDPR’s “right to erasure” so complicated in practice. Regulators wrote that right with databases in mind, where deletion is a discrete operation: find the record, remove it. Applied to a trained model, deletion becomes philosophically murky. You can delete the raw training data. You can delete your account. The model itself is another matter.
OpenAI’s response to the Italian regulators illustrates the tension well. The company added an opt-out mechanism and a form to request data deletion. What it couldn’t promise was that deleting your data would change the model’s behavior, because retraining a large language model from scratch to exclude specific data is not a practical operation. The compute costs alone are prohibitive. Even techniques like machine unlearning (a genuine area of research aimed at selectively removing training influence) remain experimental and don’t yet work reliably at the scale these companies operate.
So when you delete your OpenAI account today, here’s what actually happens: your stored conversations are deleted from OpenAI’s servers (assuming you haven’t opted out of training, or if you request it explicitly). Your email, billing details, and profile are removed. Any future data collection from you stops. The model that was trained on your prior interactions? It stays exactly as it is.
Why It Matters
For most users, this is probably fine. Your chat about meal planning or debugging a Python script is unlikely to have meaningfully shaped GPT-4’s worldview. The concern scales with sensitivity.
Consider the categories of data people routinely share with AI assistants: medical symptoms, relationship problems, legal questions, business strategies, source code for unreleased products. Many users treat these tools like a private journal or a trusted advisor. The assumption is that deletion means deletion.
It doesn’t, quite.
For businesses, the stakes are higher. Teams using AI tools with default training-on settings may inadvertently be contributing proprietary information to shared models. A developer who pastes internal API specs into a chat window, a product manager who shares an unannounced roadmap while brainstorming, a lawyer who describes a case while drafting a letter: these interactions may shape a model used by millions of others, including competitors.
This is exactly why enterprise tiers at OpenAI, Anthropic, and Google typically default to training-off configurations. The feature exists. Most free-tier users don’t know they need to find it.
What We Can Learn
The Italian case didn’t resolve the fundamental technical problem. What it did was force a company to make its practices legible. That’s useful, even if incomplete, and it points toward what individuals and organizations can actually do.
Check your training settings before you need to. Every major AI platform now has a setting that controls whether your conversations are used for training. On ChatGPT, it’s under Settings > Data Controls. On Claude (via claude.ai), Anthropic states that free-tier conversations may be used for training unless you opt out. Find the toggle before you share anything sensitive, not after.
Treat AI chat windows like email, not like a private notebook. This is a useful mental model. You wouldn’t email your lawyer’s confidential strategy memo to a generic web service. Apply the same standard to AI tools. If you’d hesitate to put it in an email, hesitate here too.
If you’re building AI products, make your data lifecycle explicit. The Garante penalized OpenAI not just for what it did, but for failing to make it clear. Users had no obvious way to understand what was collected, when, or how to ask for it to stop. If your product uses training data, your privacy policy and onboarding should say so plainly. This is both legally prudent and, frankly, the right thing to do.
Don’t wait for regulation to catch up to the technology. The right to erasure as written doesn’t map cleanly onto how these models work. Regulators know this, researchers are working on it, and legislation is moving. But the gap between what users expect and what’s technically true exists right now. Acting as though the gap doesn’t exist, whether you’re a user or a builder, is how you get caught on the wrong side of it.
Machine unlearning research is genuinely promising. Papers from Google, MIT, and others have explored methods for selectively degrading a model’s memory of specific training examples. Some approaches work reasonably well on smaller models. None yet work reliably at the scale of a deployed frontier model. This will change, probably within a few years. Until it does, the honest answer is that deletion as users understand it and deletion as engineers can implement it are not the same thing.
The Italian regulators were right to push. OpenAI was right to respond. Neither side fully solved the problem. That gap, between what we can promise users and what the technology can actually enforce, is where the interesting and important work is happening right now.
