RAG in Production for B2B Software
A demo RAG chat that answers from a PDF dump is not a B2B product. Production retrieval-augmented generation must respect tenant boundaries, cite sources operators can verify, survive stale indexes, and fail closed when retrieval confidence is low. Enterprise buyers ask how answers are grounded, who can see which documents, and what happens when the model invents a policy that never existed. This guide covers indexing, retrieval quality, multi-tenant isolation, evaluation, cost control, and audit for custom SaaS and internal tools. It pairs with agentic workflows when RAG is a tool inside a larger graph, and with multi-tenant architecture and audit logging when knowledge becomes a compliance surface.
When RAG fits B2B products (and when it does not)
RAG fits when answers must combine a language model with private, changing corpora: policies, manuals, tickets, contracts, product catalogs, SOPs. It beats fine-tuning when content updates weekly and you need citations. It does not fit as a substitute for transactional truth. Order status, inventory counts, and invoice totals belong in APIs and databases, not vector search over last night's export. If the job is multi-step action (create ticket, post ERP, wait for approval), you need an agentic workflow that may call RAG as one tool, not a single chat completion.
- Good fit: policy Q&A with citations per tenant
- Good fit: search across manuals plus structured filters
- Poor fit: inventing ERP posts from prose
- Poor fit: unsupervised answers on regulated advice without review gates
Indexing and chunking that survive real documents
Chunking is product work. Tables, headers, and procedure steps break naive fixed-size splits. Prefer structure-aware chunks: section + heading path, page number, document version, effective date. Store metadata that retrieval and UI both need: tenant_id, site_id, ACL labels, source URI, checksum, ingested_at. Re-index on publish or ACL change, not only on a nightly cron. Dual-write or event-driven ingestion from the system of record beats batch scrapes that silently drift. Treat index lag as an SLO alongside API latency.
- Keep parent document ID so UI can show full context
- Version embeddings when you change model or chunk strategy
- Delete or tombstone chunks when documents are revoked
- Separate experimental indexes from production retrieval paths
Retrieval quality: hybrid search, rerank, and filters
Pure vector search fails on part numbers, SKUs, and exact clause titles. Hybrid lexical + dense retrieval, then a reranker, is the usual production stack for B2B corpora. Always apply hard filters before similarity: tenant, ACL, language, document type, effective date. Never retrieve first and filter later if leakage is a risk. Return enough context for grounding, not a dump of the nearest fifty chunks. Cap tokens, diversify sources, and pass explicit 'no relevant source' when scores are weak.
Tenancy and ACL on the retrieval path
Multi-tenant RAG fails catastrophically if embeddings from one customer can surface in another's answer. Isolation must live in the query filter and in index partitioning strategy, not only in the chat UI. Map roles the same way as the rest of the product: plant viewer vs. HQ admin vs. partner. Document ACL changes must invalidate or re-tag chunks immediately. Align with SSO and identity so the retrieval principal is the authenticated user, not a shared service account without scope.
- Enforce tenant_id in every retrieval query
- Log denied retrieval attempts for security review
- Test cross-tenant leakage in CI, not only in demos
- Treat contractor staging data as a real tenancy boundary
Grounding, citations, and refusal behavior
Operators trust answers they can open. Require citations with document title, section, and link. Prefer extractive quotes for critical policy claims. Instruct the model to refuse or ask for clarification when retrieval is empty or conflicting. A confident wrong answer is worse than 'I could not find this in your knowledge base'. For regulated or high-stakes domains, add human review before the answer leaves the system, or restrict RAG to draft-only modes.
Evaluation harness before the pilot expands
Build a golden set of questions with expected sources and acceptable answers per tenant persona. Score retrieval hit rate, citation correctness, and answer faithfulness separately. Regress on every change to chunking, embedding model, prompt, or reranker. Pilot feedback without a harness becomes vibes-based tuning. Pair with testing strategy: unit tests for filters and ACL, integration tests for ingestion, offline eval for quality, and limited online shadow traffic before cutover.
- Track 'unsupported claim' rate as a first-class metric
- Include adversarial and empty-corpus cases
- Review failures with domain experts, not only engineers
- Gate releases on eval thresholds like any other quality bar
Cost, latency, and operations
Token and embedding spend grow with corpus size, chat volume, and naive 'stuff more context' prompts. Cache embeddings, reuse retrieval for follow-ups carefully, and budget max context per request. Measure p95 latency of retrieve + generate separately. Users blame the product, not the vector DB, when a plant floor lookup takes twelve seconds. Observe retrieval emptiness, tool errors, and model refusals via observability. Alert on sudden citation failure spikes after reindex jobs.
Audit trails for AI answers
Enterprise security questionnaires ask what the model saw. Persist prompt version, retrieved chunk IDs, filters applied, model ID, and final answer for support and dispute windows. Retention policies must cover AI traces like other audit logs. Redact secrets from stored prompts; never log raw API keys or full PII dumps 'for debugging'.
Scoping a RAG MVP without boiling the ocean
Start with one corpus, one persona, one workflow (e.g. SOP lookup for operators). Prove citation trust and ACL before adding five connectors and an agent swarm. Use MVP prioritization and technical discovery to map which documents are authoritative and who owns updates. Budget time for data hygiene; dirty SharePoint dumps dominate cost more than model choice.
Next steps
Pick ten real operator questions and measure whether retrieval returns the right source today. If not, fix indexing and ACL before prompt engineering. See agentic AI and LangGraph, other resources, case studies, book a call, or contact if you need a production RAG review before an enterprise pilot.
FAQ
Do we need a vector database for B2B RAG?
Often yes at scale, but start with whatever your stack supports if the corpus is small and filters are strict. The hard problems are ACL, evaluation, and ingestion freshness, not the logo on the vector store.
Is fine-tuning better than RAG for company knowledge?
For frequently changing private documents with citations, RAG wins. Fine-tuning helps style, domain language, or narrow classification. Many products combine both later; do not start there.
How do we prevent hallucinations in production?
Ground answers in retrieved sources, refuse when retrieval is weak, cite evidence, evaluate faithfulness, and keep humans in the loop for high-stakes outputs. Prompt wording alone is not enough.
Can contractors ship RAG safely?
Yes if acceptance includes ACL leakage tests, golden-set eval thresholds, audit of retrieved IDs, and runbooks for reindex and rollback. See contractor hiring practices for quality gates beyond a shiny demo.