Should AI agents use RAG or query the database directly?
Updated · DBShifts
Short answer
Use retrieval-augmented generation (RAG) for unstructured documents, and governed queries for live structured data. Embedding database rows into a vector store goes stale as rows change, cannot answer totals or joins reliably, and copies regulated data into a second system. Querying the source keeps answers current and data under existing controls.
Key facts
- RAG retrieves passages that look similar to the question; databases answer exact questions.
- A vector index of database rows is out of date the moment a row changes.
- Totals, counts and joins cannot be computed reliably from a handful of retrieved chunks.
- Copying regulated data into a vector store creates a second system that needs the same protection.
- Many real assistants use both: RAG for documents, governed queries for records.
When RAG is the right tool
Policies, manuals, contracts, tickets as prose — content where the answer is a passage, and similarity search finds it. RAG is good at that and nothing here argues against it.
When the database is the right tool
'How many orders shipped late this week?' and 'Which customer spent the most?' need exact, current numbers. A query answers them; a retrieved chunk can only guess, and it guesses from yesterday's copy.
The governance question doesn't go away
Either way the AI system reads your data. With RAG, the copy in the index needs masking, access control and deletion handling of its own. With direct queries through a gateway, the existing controls apply at the source.
Checklist
- 1
Use RAG for documents and long-form text
- 2
Query the source for records, numbers and anything that changes
- 3
Put a governed gateway between agents and the database
- 4
If you do embed records, mask personal data before embedding
- 5
Plan deletion: a removed customer must disappear from every copy
Common mistakes
Embedding whole tables 'to be safe'
It feels contained, but it creates an unmasked, unaudited, eventually stale copy of production data.
Asking RAG for totals
Similarity search returns some matching rows, not all of them. Sums and counts built on it are wrong in ways that look plausible.
Forgetting deletion
Data removed from the database often survives in the vector index until someone rebuilds it.
How DBShifts does it
- DBShifts Agent Gateway gives agents governed, read-only queries against the live source, so answers are current and existing masking and limits apply.
- Agents can aggregate and follow real foreign keys, which covers the totals and joins retrieval cannot.
- When the text an agent needs already lives in the database — tickets, notes, articles — gateway search indexes retrieve it in place: full-text search on PostgreSQL and MySQL, or nearest-neighbour search on a pgvector column with an embedding the caller supplies.
- Search results pass the same masking, row filters, budgets and injection shield as any other read, and a masked column can never be searched, because a match would reveal what the mask hides.
Questions people also ask
Can I use RAG and a database gateway together?
Yes, and many assistants should: documents through retrieval, records through governed queries.
Can I do retrieval without copying data into a vector store?
If the text and any embeddings already live in your database, yes: search them where they are, through the same access controls as every other read, so there is no second copy to secure and delete from.
Isn't querying production risky?
Unrestricted querying is. Read-only access through allowlists, masking, row caps and budgets is how you make it safe.
Related questions
Give AI agents data access you can defend
Scoped tokens, PII masked on the server, every request audited. Free to start.
Start free18 answers in this series. See all.