RAG on sensitive data: 7 checks before going to production
A checklist for RAG on internal data, from permissions at retrieval time and prompt injection to automated testing of LLM outputs.
A RAG prototype takes an afternoon: documents into a vector database, an LLM in front, done. With sensitive data, the path to production is much longer. These are the seven points I check before a RAG system on internal data goes live.
1. Enforce permissions at retrieval time
The most important principle: the model may only see what the requesting user may see. Permissions from the source systems therefore belong in the index as metadata and are applied as a filter on every search. An instruction in the prompt such as “don’t use confidential documents” is not access control.
2. Classify data before it enters the index
Not every document belongs in a RAG index. Decide up front which protection classes exist, which sources are connected and who signs off. That saves painful removal from index and logs later.
3. Choose model and hosting by protection level
Model choice follows from data classification: a cloud service such as Azure OpenAI in an EU region, a model on your own infrastructure on Google Cloud, AWS or Azure, or a locally run model, for example with Ollama. Contracts, data residency and the provider’s logging behaviour belong in this decision.
4. Plan for prompt injection
Documents in the index are input that the model may mistake for instructions. Limit what the system may do with an answer, separate system instructions clearly from retrieved content, and treat tool calls from the model like unvalidated user input.
5. Test outputs automatically
Every change to prompt, model, chunking or index can make answers worse. An automated test suite with fixed questions, expected sources and evaluation criteria makes those changes measurable. In a banking environment, I built a dedicated testing suite for LLM outputs for exactly this.
6. Logging and observability with privacy in mind
You need insight into queries, retrieved documents, latency and cost. At the same time, exactly these logs contain sensitive data. Define what is stored, for how long and who has access.
7. Harden the platform
A RAG system is still an application with APIs, containers and identities. Hardened containers, single sign-on via OAuth2/OIDC, least-privilege IAM and security scanning in the CI/CD pipeline apply here just as everywhere else.
Conclusion
The real work in RAG on sensitive data isn’t in the model; it’s in permissions, data flows, testing and operations. Building these points into the architecture from day one gets you to production faster and more securely.
Frequently asked questions on this topic
What is RAG?
Retrieval-augmented generation. A language model answers questions based on documents retrieved from an index at runtime, rather than from its training data alone.
Is it enough to tell the model in the prompt which data it must not use?
No. A prompt is not access control. Documents a user is not allowed to see must never reach the model’s context in the first place.
Do I have to self-host models for RAG on sensitive data?
Not necessarily. Data classification, contracts and region are what matter. Depending on the protection level, a cloud service in an EU region or a self-hosted model can fit.