Ingestion is the first quality gate
Documents arrive from many systems: drives, CRM, support, knowledge bases, databases and file uploads. The ingestion pipeline should normalize formats, extract text, preserve source identity and record version metadata. Bad ingestion creates retrieval problems that no prompt can repair.
Make ingestion idempotent so reprocessing the same source does not create duplicates. Track source timestamps, checksums and deletion events. A production system needs to know when content changed and when previously indexed content must be removed.
Chunking should follow meaning and source structure
Fixed-size chunks are simple but often destroy context. Better chunking uses headings, sections, tables, message threads or domain-specific boundaries. The goal is to preserve enough semantic context without flooding retrieval with oversized documents.
Store metadata with every chunk: document id, title, section, owner, timestamps, tenant and security labels. Metadata enables filtered retrieval and makes debugging far easier.
Permissions must be enforced before generation
RAG can accidentally become a data-exfiltration layer if retrieval ignores access control. Apply user and tenant permissions during retrieval, not only after the answer is generated. The model should never receive content the user is not allowed to see.
This usually means propagating source ACLs into the index and filtering candidate chunks before they reach the reranker or LLM. Permission logic should be testable independently of the model.
Use hybrid retrieval and reranking
Dense embeddings are useful, but lexical search still matters for IDs, product names, error codes and exact terminology. Hybrid retrieval combines semantic and keyword signals, then reranks a smaller candidate set.
Measure recall on known-answer datasets. If the right evidence is not retrieved, changing the prompt is the wrong fix. Retrieval quality should be treated as its own engineering subsystem.
Evaluate groundedness and answer usefulness separately
A response can be factually grounded but still unhelpful, or useful but unsupported. Evaluation should score citation correctness, answer completeness, retrieval recall, refusal behavior and unsupported claims.
Create representative question sets from real user workflows. Track metrics across prompt, embedding and model changes so upgrades are evidence-driven rather than subjective.
Trace every answer back to evidence
Production observability should show the query, rewritten query, retrieved chunks, reranker scores, model version, prompt version, citations, latency and user outcome.
This trace lets engineers answer the critical question after a bad response: was the source missing, was retrieval weak, did the reranker fail, or did the model ignore good evidence? Without that separation, teams tune the wrong layer.