The demo goes well. Someone types a question about a policy, a contract or a procedure, and the assistant answers in a paragraph with a citation. Leadership is impressed. Two weeks later the same assistant tells a customer service agent that a product still ships to a region it left last quarter, quotes a clause from a superseded contract, and shows a junior analyst a salary band they were never meant to see. Nobody touched the model. The model was never the problem.
Retrieval-augmented generation, or RAG, is a simple idea: before the model answers, fetch the documents that matter and hand them to it. The quality of the answer is bounded by the quality of the fetch. Most pilots spend ninety percent of their effort on the prompt and the interface, and ten percent on the fetch. Production inverts that ratio. Here is where the ten percent breaks, in the order it usually breaks.
Week one: freshness
A demo runs on a snapshot. Someone exported a folder of PDFs, loaded them once, and the pilot worked because the world stood still for the length of the demo. Production is a moving target: policies get amended, price lists change, a product is retired, a clause is renegotiated. If the pipeline that loaded the documents does not run again, the assistant is confidently wrong within days, and it is wrong with a citation, which makes the error more convincing, not less.
What fixes it is unglamorous: a pipeline that knows where each document came from, checks for changes on a schedule that matches how fast that source moves, re-embeds what changed, and retires what was deleted. Every source needs a freshness target written down, and something needs to alert a human when it is missed. The question to ask your team is not "is the data loaded" but "when a document changes at the source, how long until the assistant knows".
Week two: chunking and retrieval quality
Documents are cut into pieces before they are indexed, because the model can only read so much at once. How you cut them decides what can be found. Cut a contract every 500 words and the definition of a term lands in one piece while its use lands in another; the assistant retrieves the use, never sees the definition, and invents one. Cut a procedure by page and a step that spans the page break disappears.
There is no universal right answer, which is exactly why this is engineering rather than configuration. Contracts want clause-aware splitting. Procedures want step-aware splitting. Tables want to stay tables. And none of it can be judged by reading a few answers. It has to be measured against a labeled set: fifty or a hundred real questions with the passages that should be retrieved, scored every time the chunking, the embedding model or the index changes. Teams that skip the labeled set are tuning by anecdote, and anecdote always favors the last thing that went wrong.
Week three: access control
This is the one that ends pilots. Enterprise documents carry permissions: this folder is for HR, that contract is for the deal team, those board minutes are for six people. A retrieval index flattens all of it into one searchable pool. Unless permissions travel with each chunk and are enforced at query time against who is asking, the assistant becomes the most efficient data leak the company has ever deployed.
The fix is document-level and often row-level permission metadata, captured at ingestion from the source system and checked on every retrieval before anything reaches the model. That means the pipeline has to understand the source system's security model, which is different for SharePoint, a document management system, a CRM and a data warehouse. It is real work, and it is the work that lets a compliance officer sign off.
What to budget
Across the pilots we have seen turn into platforms, the data work, meaning ingestion, freshness, chunking, evaluation and permissions, consumes as much effort as the application work, and frequently more. A budget that assumes otherwise is a budget for a demo.
The role that owns this is a Data Engineer who has done unstructured and vector pipelines before, paired with an AI Engineer on the application side and, once the system is live, an LLMOps Engineer to keep the evaluation harness running. Most organizations already have strong data engineers. Fewer have data engineers who have built for retrieval, where the quality metric is "did the right passage come back" rather than "did the table load".
Before you call the pilot done
- Every source has a written freshness target and an alert when it is missed.
- A labeled set of real questions exists and is run automatically on every change to chunking, embeddings or the index.
- Retrieval precision on that set is reported weekly, and someone owns the number.
- Permissions are captured at ingestion and enforced at query time, and a test proves a user cannot retrieve what they cannot open at the source.
- The cost per thousand queries is known, and the estimate for year-two volume has been shown to whoever pays for it.
- The assistant has a designed "I do not know" path, and it is triggered more often than never.
A pilot that passes this list is not a pilot any more. It is the first version of a system, and it can be handed to the people who will run it.