Start with the failure modes
Most RAG demos look great on curated PDFs. Production breaks on stale documents, overlapping chunks, missing ACLs, and unanswered “I don’t know” cases. Define those failure modes before you pick a vector database.
Chunk for questions, not pages
Chunk size should follow how users ask questions. Meeting notes, SOPs, and API docs need different strategies. Add metadata early—tenant, department, freshness—so retrieval can filter before similarity search.
Evaluate every week
Keep a golden set of questions with expected citations. Run it in CI when prompts or embeddings change. Without regression tests, every “improvement” is a gamble.
Budget tokens like money
Cap context windows, cache frequent retrievals, and log cost per successful answer. Latency and spend are product features for enterprise buyers.