Start with the failure modes
Most RAG demos look great on curated PDFs. Production breaks on stale documents, overlapping chunks, missing ACLs, and unanswered “I don’t know” cases. Define those failure modes before you pick a vector database. Write them down as product requirements: what happens when the contract changed last week, when two SOPs contradict, when a user from tenant A must never retrieve tenant B, and when the right answer is to refuse. If you cannot name those cases, you are buying a chatbot, not a retrieval system. VyrroTech scopes RAG as an operations tool with citations, evaluation, and a budget—not a slide with a sparkline.
Chunk for questions, not pages
Chunk size should follow how users ask questions. Meeting notes, SOPs, and API docs need different strategies. A 512-token window that worked on a blog corpus will shred a table-heavy policy PDF. Add metadata early—tenant, department, freshness, document type—so retrieval can filter before similarity search. Overlapping chunks without titles produce citations nobody can audit. Keep a human-readable source pointer (page, section, URL) on every chunk. If legal or clinical corpora are in scope, treat PII redaction as a pipeline stage, not a prompt instruction. Data residency for UAE, KSA, UK, and US buyers is a retrieval and logging design problem, not a checkbox on the model vendor’s marketing site.
Evaluate every week
Keep a golden set of questions with expected citations. Run it in CI when prompts or embeddings change. Without regression tests, every “improvement” is a gamble. Include must-refuse questions so the system is scored on silence as well as answers. Track citation precision separately from fluency; a confident wrong paragraph with a plausible footnote is a worse failure than a short “I don’t know.” When the corpus updates, re-run the set before you celebrate a new embedding model. We treat evaluation harnesses as part of delivery, the same way we treat tests on a billing API.
Budget tokens like money
Cap context windows, cache frequent retrievals, and log cost per successful answer. Latency and spend are product features for enterprise buyers. Unbounded “stuff the top 20 chunks into the prompt” designs look smart in a notebook and fail in a 2-hour SLA support queue. Cache at the retrieval layer for repeated questions; do not log raw documents into a third-party trainer. Put a hard timeout and a fallback (search UI, human ticket) on the path so a model outage is not a silent empty page. If unit economics cannot name cost per resolved question, you are not ready to put RAG on a customer-facing SLA.
Access control is retrieval, not a UI toggle
Never retrieve first and filter later in the prompt. Tenant and role filters belong in the query. Document ACLs that live only in the application database will leak the first time a job or notebook uses the vector index directly. We derive tenancy from the session, the same rule as our multi-tenant SaaS work. Audit who asked what, which chunks were returned, and whether the answer was shown or refused. That timeline is how you investigate an incident instead of arguing with a screenshot.
How we ship RAG with clients
Production only: retrieval with citations, evaluation in the pipeline, token and latency budgets, and human-in-the-loop for high-risk actions. Models (Gemini or otherwise) sit behind your corpus and guardrails. If the brief is a weekend demo on public PDFs, we will say so and not dress it up as an enterprise knowledge agent. For a scoped build, bring the top 20 questions operators actually ask, the systems of record, and the access model. We will tell you which of those questions RAG can answer and which still need a human or a deterministic tool.
A realistic first production slice
Do not start with “all company knowledge.” Pick one team, one corpus that already has owners, and the ten questions that cost the most time today. Instrument refuse rates, citation click-through, and cost per resolved question before you add a second department. Freshness is an operations job: someone must own document updates or the golden set will slowly lie. We would rather ship a narrow assistant that operators trust than a portal that hallucinates across SharePoint. That is the same delivery standard as the rest of VyrroTech AI work: named lead, written scope, 100% IP, 90-day warranty, 2-hour response on business days. If your corpus cannot be permissioned, RAG is the wrong product until it can.
