How to Protect PII in LLM Prompts: A Developer’s Guide

Why Every Prompt is a Potential Data Breach

When you send a prompt to ChatGPT, Claude, Gemini, or any commercial LLM API, that data is processed on servers you do not control. Providers log prompts for abuse monitoring and safety evaluation. Consumer chat tools may use your conversations for training unless you opt out; API and Enterprise plans from OpenAI, Anthropic, and Google do not train on your data by default. Your enterprise API key does not make your prompts private. A Business Associate Agreement (BAA) shifts legal liability, but it does not prevent the data from flowing to the provider’s infrastructure.

The practical risk is significant. A Cyberhaven study found that 11% of data pasted into ChatGPT by enterprise users includes sensitive information — customer PII, financial data, source code, and internal documents. For a healthcare organization, a legal firm, or any company handling regulated data, this is not a theoretical risk. It is a daily occurrence.

The 3 Ways PII Leaks Through AI Prompts

1. Training Data Contamination

If your LLM provider uses conversation data for training (consumer chat tools often do by default unless you opt out), real PII you included in prompts can end up embedded in the model’s weights. Cases have been documented where models reproduce fragments of real personal data that appeared in their training corpus. Once in a model’s weights, this data cannot be removed without full retraining.

2. Prompt Logging by AI Providers

All major LLM providers log prompts for abuse monitoring, safety research, and service improvement. These logs are retained for varying periods. Even with data processing agreements in place, your prompts are visible to provider employees with access to monitoring infrastructure. For HIPAA-covered entities, the mere transmission of PHI to an external system without adequate safeguards is a HIPAA violation, regardless of whether the data is subsequently misused.

3. Output Exposure in Conversation History

LLM conversations are frequently shared. A developer pastes an LLM response into a Slack message. A manager screenshots a conversation for a report. A support agent copies an AI-generated response into a ticket system. If the original prompt contained PII, the response may echo or reference it, and that response now propagates through your organization’s communication systems — often with weaker security controls than your primary data stores.

Solution 1 — MCP Server (For Developers)

The Model Context Protocol (MCP) is an open standard for connecting AI assistants to tools and data sources. anonym.legal provides an MCP server that gives your AI assistant tools to detect and anonymize PII and to restore tokenized values. The assistant calls these tools when needed; the server is not a proxy that automatically intercepts every prompt.

To add the anonym.legal MCP server to Claude Desktop or another MCP-compatible AI assistant:

Once configured, the MCP server exposes nine tools to the AI assistant:

• analyze text — Identify PII entities without changing the text
• anonymize text — Detect and replace PII, optionally as tokens stored in a tokenization session
• detokenize text — Restore original values in tokenized text
• list sessions / delete session — Manage tokenization sessions and their token mappings
• redact image — Redact or analyze PII in an image using OCR
• list presets, get balance, estimate cost — Presets, token balance, and cost before processing

The MCP server runs on anonym.legal’s servers at https://anonym.legal/mcp; your assistant reaches it with your API key. It protects the text the assistant passes through its tools; anything you type directly into the chat reaches the model before any tool runs.

Solution 2 — Chrome Extension (For Non-Technical Users)

For users who work directly in the ChatGPT, Claude, or Gemini web interface, the anonym.legal Chrome Extension (available on paid plans) provides transparent PII protection without requiring any API integration.

When a user types or pastes text into the AI chat interface, the extension:

1. Detects PII entities in real-time using the anonym.legal detection engine (273 entity types)
2. Replaces PII before the text is submitted
3. Optionally de-anonymizes the AI’s response before displaying it to the user

The key differentiator is reversible encryption. Unlike simple replacement (“John Smith” → “PERSON_1”), AES encryption preserves the ability to recover the original value. If the AI produces a response that references “PERSON_1” and you need to share it with a colleague who knows the real person, the extension can silently decrypt the response for authorized users while maintaining the encrypted form in transit and storage.

This is particularly important for legal and healthcare workflows where the AI output must be actionable. A legal assistant using AI to draft a settlement letter cannot use a letter that says “Dear PERSON_1” — but they also cannot send the AI a letter containing the real client’s personal details. Reversible encryption solves both problems simultaneously.

Solution 3 — REST API (For Application Builders)

If you are building an application that calls an LLM on behalf of users — a customer service bot, a document summarization tool, a code review assistant — you need to sanitize user-provided data before it reaches the LLM. The anonym.legal REST API provides a drop-in preprocessing step for your LLM call chain.

The De-anonymization Workflow

The encrypt/decrypt workflow is what separates anonym.legal from simple redaction tools. Here is how a typical session works:

1. User input: “Draft a letter to Maria Gonzalez (maria.g@acme.com) about her account #847391.”
2. Encrypted by extension: “Draft a letter to [AES:maria-001] ([AES:email-001]) about her account [AES:acct-001].”
3. LLM processes the request using encrypted tokens as entity references
4. LLM response: “Dear [AES:maria-001], Regarding your account [AES:acct-001]…”
5. Extension decrypts for authorized user: “Dear Maria Gonzalez, Regarding your account #847391…”

The LLM never saw real PII. The response is fully actionable for the authorized user. The audit log shows only encrypted tokens. The original values can only be recovered by someone with the session key.

What Zero-Knowledge covers here

In anonym.legal, Zero-Knowledge covers only custom patterns you store: they are encrypted in your browser, and the server keeps only ciphertext. Encryption keys are derived on your device with Argon2id. The sign-in is not zero-knowledge, and neither is processing: the text is analysed on our servers in Germany, and the key of the encrypt operator is sent with the request. Where the text may not be read by anyone else at all, anonym.plus does the same work on your own machine.

Where the key mapping lives matters: if the anonymization service holds it, a breach of that service can expose the pseudonym mappings. On anonym.plus, the mapping never leaves your device. On anonym.legal, a tokenization session mapping is held server-side (24 hours by default, up to 30 days if you choose persistent) unless you use end-to-end mode.

Compliance Impact: GDPR, HIPAA, EU AI Act

This workflow addresses multiple regulatory requirements simultaneously:

• GDPR Article 25 (data protection by design): Technical measures applied before data leaves your environment
• GDPR Article 28 (processor agreements): Encrypted tokens reduce what the LLM provider can read, but an Article 28 processor agreement may still be advisable — consult counsel for your specific setup
• HIPAA Minimum Necessary: Only the minimum information needed for the AI task is transmitted — and even that is encrypted
• EU AI Act Article 10: Inference pipelines for high-risk AI systems can demonstrate technical PII minimization

Conclusion: Anonymize First, Then Query

The productivity benefits of AI are real and significant. The data protection risks are equally real. The solution is not to avoid AI — it is to anonymize before querying. With MCP Server integration for developers, Chrome Extension for direct users, and REST API for application builders, anonym.legal provides tools to anonymize before querying across these workflows.