Enterprise RAG: when it works and what it costs

Key takeaways

• ✓RAG (retrieval-augmented generation) connects a language model to your own documents and data at query time, so answers are grounded in your content rather than the model's training data alone.

• ✓It works well when your organisation has clean, well-structured knowledge that a general-purpose model cannot access: internal policies, product documentation, support histories, regulatory guidance.

• ✓RAG is frequently oversold. If your underlying data is poorly organised, inconsistent, or ungoverned, RAG will surface that mess at scale.

• ✓The real costs are rarely the API or hosting fees. Expect to spend most of your budget on data preparation, retrieval tuning, evaluation, and the ongoing work of keeping content current.

• ✓Most RAG projects that fail do so at the data layer, not the model layer. Getting that right before you build is what separates a useful system from an expensive proof of concept.

What is enterprise RAG and how does it actually work?

Retrieval-augmented generation (RAG) is a technique that connects a large language model (LLM) to your own data sources, so the model answers questions using your documents rather than only its training data. The model does not memorise your content. Instead, at the moment a user asks a question, a retrieval system pulls the most relevant chunks of text from your knowledge base and hands them to the model as context. The model then generates an answer grounded in that retrieved material.

The basic pipeline has four moving parts:

• Ingestion. Your source documents, whether PDF policies, SharePoint pages, Confluence wikis, or database records, are broken into chunks and converted into numerical representations called embeddings. These embeddings capture semantic meaning, so "termination clause" and "end of contract" land close together in the index.

• Storage. The embeddings are stored in a vector database (common options include Pinecone, Weaviate, pgvector in Postgres, and Databricks Vector Search). This is where your retrieval queries run.

• Retrieval. When a user submits a query, it is also converted to an embedding. The vector database finds the chunks most similar to the query and returns them as context.

• Generation. The LLM receives the query plus the retrieved chunks and writes a response, ideally citing the source material.

That is the clean version. In practice, enterprise RAG implementations add layers: document pre-processing pipelines to handle scanned PDFs and inconsistent formatting, re-ranking steps to improve retrieval quality, chunking strategies tuned to document type, and guardrails to prevent the model from straying beyond what was retrieved.

Why enterprises reach for RAG

General-purpose LLMs know nothing about your internal policies, your product catalogue, or your contracts. Fine-tuning a model on that data is expensive and slow to update. RAG lets you keep the model static and update the knowledge base instead, which suits organisations where documents change regularly.

The reason enterprises reach for RAG specifically, rather than simply fine-tuning a model on internal data, comes down to freshness and cost. Fine-tuning bakes knowledge into model weights. If your pricing document changes next month, you retrain. With RAG, you update the vector index. For document-heavy use cases such as legal, compliance, HR policy, or technical support, that difference matters a great deal.

What RAG does not do is make a weak model smart, fix poorly organised source data, or guarantee accuracy. The quality of the answers is directly bounded by the quality of what gets retrieved, and retrieval quality depends heavily on how the data was prepared. That constraint shapes almost every challenge teams run into when they move from prototype to production.

When does enterprise RAG implementation make sense?

RAG delivers genuine value when three conditions line up: the organisation holds knowledge that no general-purpose model was trained on, the cost of a wrong answer is high, and that knowledge changes often enough that periodic fine-tuning would become a maintenance burden.

You have a private knowledge base that matters

The clearest RAG use case is a corpus of internal documents that staff currently search by hand. Think technical product manuals, regulatory submissions, internal policy libraries, contract repositories, or years of accumulated project documentation. A general-purpose model like GPT-4 has no access to any of it. RAG solves this by retrieving the relevant passages at query time and feeding them to the model as context, so the answer is grounded in your actual documents rather than in training data.

If your team spends meaningful time hunting through SharePoint, Confluence, or a shared drive to answer recurring questions, that's a reasonable signal that RAG is worth scoping properly.

The cost of hallucination is non-trivial

Base language models hallucinate. Not constantly, but enough to matter in high-stakes contexts. If a model invents a clause in a contract, misquotes a regulatory threshold, or cites a policy that was superseded two years ago, the downstream consequences can be significant. RAG reduces (not eliminates) hallucination risk by anchoring responses to retrieved source material, and it lets you implement citation so the user can verify the answer against the original document.

This matters most in legal, compliance, finance, and engineering contexts where incorrect information has real consequences. It matters less in creative or exploratory tasks where precision is not the point.

Retrieval grounds the answer, it does not guarantee it

RAG reduces hallucination by giving the model real source material to work from. It does not remove the possibility of error. High-stakes outputs still need a human review step, particularly where the retrieved documents are ambiguous or contradictory. See how to build a verification protocol for high-stakes requests for a practical approach.

Your knowledge base changes frequently

Fine-tuning, the alternative approach where you retrain or further-train a model on your data, produces a model that knows your content at the point of training. If that content updates regularly, you are re-training regularly, which is expensive and operationally complex. RAG sidesteps this because the retrieval layer reads from a live document store. Update the document, and the next query gets the updated answer.

A compliance team that maintains regulatory guidance updated quarterly, or a product organisation with documentation tied to a release cycle, will find RAG easier to keep current than fine-tuning.

A few qualifying questions worth asking first

Before committing to a RAG build, it helps to honestly answer these:

• Is the knowledge genuinely proprietary, or is it information a well-prompted general model already handles well?

• Are your source documents well-structured and consistently formatted, or are they a mixed-quality archive that would produce unreliable retrieval?

• Do you have the data engineering capability in-house to build and maintain a vector database (the index that powers retrieval), or will this need external support?

• Is there a specific workflow or user group with a clear pain point, or is the project more exploratory?

RAG projects that start with a concrete use case and a defined user group are considerably more likely to deliver value than those that begin with the technology and search for applications. This point comes up repeatedly in the context of building and prioritising an AI use case backlog: the question is never "can we do RAG?" but "does RAG solve a problem worth solving?"

When is RAG the wrong tool?

RAG solves a specific problem: giving an AI model reliable access to information it was not trained on. If that is not your problem, you are adding complexity for no gain.

A few situations where RAG is genuinely the wrong choice:

Your data volume is small. If the information you want the model to use fits comfortably in a prompt, just put it there. Direct context injection is simpler, cheaper, and easier to debug than standing up a retrieval pipeline. RAG starts earning its complexity when you have more content than a single prompt can hold, not before.

Your data is already well-structured. RAG works by searching unstructured or semi-structured text. If your source of truth is a relational database with clean schema, consistent fields, and reliable keys, a direct database query will almost always return a more accurate answer than a retrieval system hunting through chunked text. A finance team checking invoice status does not need RAG. They need a query.

Your data foundations are weak. This is the most common failure mode. RAG retrieves what exists. If your documents are inconsistent, outdated, duplicated, or poorly organised, the model will confidently surface wrong answers. Garbage in, garbage out applies harder here than in most data contexts, because the outputs look authoritative even when they are not. Before committing to a RAG implementation, it is worth reading through what good data literacy looks like across an organisation, because data quality is the load-bearing wall.

The real problem is process, not access. Sometimes teams reach for RAG because employees cannot find the right information quickly enough. That is often a search problem, a content governance problem, or a training problem. A modern enterprise search tool or a well-maintained SharePoint index may solve it faster and at a fraction of the cost. Similarly, if the goal is to automate a decision or a workflow rather than answer a question, you may be looking at an agent or a traditional rules-based system rather than a RAG setup. The four levels of AI agent autonomy are a useful frame for working out which architecture actually fits what you are trying to do.

RAG is a retrieval problem, not a general AI fix

If you would not solve the problem by having a smart person search your document library, RAG probably will not solve it either. Be honest about what the bottleneck actually is before you commit to the architecture.

Regulatory or sensitivity constraints make retrieval risky. If your documents contain information that should only be seen by certain roles, a retrieval layer introduces access control challenges that need careful design. Getting this wrong is not a minor bug; it is a data leakage risk with real consequences. Implementing role-aware retrieval is possible, but it adds engineering time and ongoing maintenance, and it is a common underestimate in early project scoping.

None of this means RAG is overhyped. It means it is a specific tool with a specific job, and the first honest question is whether that is actually the job you need done.

What does enterprise RAG actually cost?

RAG is not expensive to prototype and genuinely costly to run at scale. That gap catches a lot of teams off guard.

Here are the main cost components to budget for.

Infrastructure and vector storage

A vector database (Pinecone, Weaviate, pgvector on Postgres, or a managed option inside Databricks) stores the numerical representations of your documents. At small scale, a managed vector store might cost AUD 150 to 400 per month. At enterprise scale, with millions of document chunks and low-latency retrieval requirements, you are looking at AUD 2,000 to 8,000 per month or more, depending on the provider and whether you self-host.

Self-hosting on AWS or Azure reduces the per-query cost but shifts the burden to your infrastructure team. That trade-off is real: you pay in engineering time rather than vendor margin.

Embedding generation

Before any document can be retrieved, it must be converted into a vector embedding. This happens once at ingestion and again each time a user query is processed. OpenAI's embedding API is cheap per token, but a large document corpus (say, 50,000 policy documents, contracts, or technical manuals) can produce a surprisingly large one-time ingestion bill. Budget AUD 500 to 3,000 for initial embedding of a mid-sized corpus, then an ongoing cost for re-embedding when documents change.

Open-source embedding models (sentence-transformers, for instance) eliminate the per-token cost but require compute to run them, usually on GPU instances.

LLM inference

This is often the biggest ongoing cost. Every user query triggers at least one call to a language model, and in more complex pipelines it triggers several. If you are using a hosted model like GPT-4o or Claude 3.5 Sonnet via API, costs scale directly with usage. A team of 200 moderate users asking ten questions per day could easily spend AUD 3,000 to 10,000 per month on inference alone, depending on context window size and model tier.

The context window is where costs escalate

RAG works by injecting retrieved document chunks into the prompt. The more chunks you retrieve, the longer the prompt, and the higher the inference cost. Getting retrieval precision right is therefore a cost-optimisation problem, not just a quality problem.

Organisations running high query volumes often move to fine-tuned smaller models or on-premise inference (via Ollama or vLLM) to cap this cost, accepting some trade-off in output quality.

Integration and build costs

This is the cost that most estimates ignore. Connecting a RAG pipeline to your existing systems (SharePoint, Confluence, a document management platform, a CRM) requires real engineering. A clean greenfield integration might take two to four weeks of a senior engineer's time. A messy legacy environment with inconsistent document formats, restricted APIs, and access-control requirements can take three to five months.

At AUD 150 to 250 per hour for a contract data or ML engineer in Australia, an integration phase alone can run AUD 50,000 to 150,000 for a complex deployment.

Ongoing maintenance

Documents change. Models improve. Retrieval quality degrades as the corpus grows unless the chunking and indexing strategy is actively maintained. Budget for:

• Re-ingestion when source documents are updated or added

• Periodic re-evaluation of retrieval quality (someone has to check that the system is still surfacing the right content)

• Model upgrades when the underlying LLM is deprecated or a better option becomes available

• Monitoring and alerting infrastructure so you know when the pipeline is returning poor results

A realistic ongoing maintenance load for a production RAG system is 0.2 to 0.5 FTE per year, depending on corpus size and query volume.

A rough total

A mid-market deployment covering one well-scoped use case (say, internal policy search for a 300-person organisation) might land at AUD 60,000 to 120,000 to build and AUD 6,000 to 15,000 per month to run. Enterprise deployments with multiple corpora, high query volumes, and tight security requirements cost more.

None of this is prohibitive if the use case justifies it. But it does mean that a RAG project pitched as "cheap because we're just calling an API" has probably not been fully costed.

What are the common failure points in RAG projects?

RAG implementations fail in predictable ways. Most of them have nothing to do with the underlying model.

The retrieval step is weaker than it looks

The model can only answer well if the retrieval step finds the right content. This sounds obvious, but it is the most common place things go wrong. Teams focus on the generation (the visible output) and underinvest in the indexing and retrieval pipeline that feeds it.

Two specific problems show up repeatedly. The first is poor chunking: splitting documents into segments that are too large, too small, or cut across logical boundaries. A policy document split mid-sentence, or a technical guide chunked so that the question and its answer land in different segments, will produce confident-sounding but inaccurate responses. The second is weak embedding quality. Embeddings are the numerical representations used to match a query to relevant content. Using a generic embedding model on domain-specific content (legal, clinical, engineering) degrades retrieval accuracy in ways that are hard to diagnose unless you are testing systematically.

Garbage in, garbage out applies doubly here

The model cannot compensate for a retrieval step that returns the wrong content. Poor chunking and weak embeddings produce plausible-sounding wrong answers, which are harder to catch than obvious errors.

The data is not as ready as assumed

Most enterprise document stores contain years of accumulated content: outdated policies sitting alongside current ones, multiple versions of the same procedure, documents with no metadata, scanned PDFs that have never been OCR-processed. When all of that goes into the index untouched, the system retrieves stale or conflicting content and presents it as authoritative.

This is a governance problem as much as a technical one. Before you index a corpus, someone needs to decide which documents are canonical, which are superseded, and who owns ongoing curation. If that work has not been done, RAG will faithfully surface your organisation's most confused thinking at high speed. A solid AI governance framework helps, but it needs to be applied to the data layer specifically, not just to model outputs.

Evaluation is skipped or done too late

Many teams build a RAG pipeline, run a few manual queries that look reasonable, and move to deployment. Without a structured evaluation process, retrieval quality, answer accuracy, and failure modes are invisible until users start complaining.

The field has developed specific evaluation approaches for RAG: measuring retrieval relevance (did the right chunks come back?), answer faithfulness (does the response actually reflect the retrieved content?), and answer correctness (is it factually right?). These can be run with a test set of known questions and answers built from your own documents. Building that test set takes time, but skipping it means you are flying blind.

The team does not have the right skills

RAG sits at the intersection of information retrieval, machine learning, data engineering, and application development. Most teams have depth in one or two of those areas, not all four. The result is a system that works well in the dimensions the team understands and quietly fails in the ones they do not.

This is not a reason to avoid RAG. It is a reason to be honest about skill gaps before scoping the project. A data engineer who has never worked with vector databases will need time to get comfortable with indexing pipelines. A developer experienced with APIs may not have intuition about why retrieval quality is degrading. These gaps are closeable, but only if they are named. The five roles every enterprise AI initiative actually needs maps this out more fully, and RAG projects are no exception to that framework.

Prompt engineering is treated as an afterthought

The prompt that wraps the retrieved content and instructs the model how to answer it matters more than most teams expect. A poorly structured prompt can cause the model to ignore the retrieved context, hallucinate beyond it, or respond in a format that does not fit the use case. Prompt design for RAG requires iteration, and the best prompts are specific to the domain and the task. Treating it as something to sort out after the pipeline is built adds rework and delay.

Frequently asked questions

How is RAG different from fine-tuning a model on our data?

RAG and fine-tuning solve different problems. RAG retrieves current documents at query time and inserts them into the model's context, so the answers reflect your latest data without retraining. Fine-tuning bakes knowledge into the model's weights during training, which suits stable domain vocabulary or a consistent output style, but it is expensive to repeat whenever data changes and it does not give the model access to specific source documents. For most enterprise document-question use cases, RAG is the right starting point. Fine-tuning becomes relevant later, often on top of a working RAG system, if you need the model to respond in a specific tone or follow a domain-specific format reliably.

What kind of data quality does RAG require before we start?

RAG exposes poor data quality immediately. If your source documents are inconsistently formatted, missing metadata, or contain contradictory versions of the same policy, the retrieval step will return noisy results and the model will either hallucinate a synthesis or hedge unhelpfully. Before building anything, audit a representative sample of your intended document corpus: check for duplicates, outdated versions, and whether documents actually contain the answers users will ask for. A strong data literacy baseline across the team matters here, because the engineers building the pipeline need to make judgements about document structure, not just write code.

How do we stop the system returning confidential documents to the wrong users?

Permission-aware retrieval is the answer, and it needs to be designed in from the start rather than bolted on later. The retrieval layer must query only documents the requesting user is authorised to see, which means your vector database or search index needs to store and enforce those permissions at the chunk level, not just the file level. This is one of the more common places RAG projects go wrong in enterprises with complex access control. The data leakage scenarios worth watching extend well beyond RAG, but retrieval systems that ignore row-level security are a significant exposure.

What evaluation metrics should we use to know if the system is working?

Two measures matter most: retrieval precision (are the retrieved chunks actually relevant to the query?) and answer faithfulness (does the generated response stay true to what the retrieved documents say, without adding claims they do not support?). Alongside those, track the rate at which users need to click through to source documents to verify answers, because a high verification rate often signals that confidence is low. Human evaluation on a sample of real queries is still the most reliable signal; automated scoring using a separate LLM as a judge is useful for volume but should not replace it. Set your baseline before you ship and measure weekly rather than waiting for complaints.

How long does a production RAG system typically take to build?

A focused proof of concept over a narrow document set can run in two to four weeks with a small team. Moving to production across a real corpus, with access controls, monitoring, evaluation pipelines, and integration into existing tools, usually takes three to six months depending on the complexity of your data environment and how much organisational approval is needed. Teams that underestimate this tend to skip the evaluation and permission layers, which creates problems that are costly to retrofit. Aligning stakeholders early on what "done" actually means is one of the highest-value things you can do before the first line of code is written; a clear AI use case backlog helps with exactly that scoping conversation.

Ready to work out whether RAG fits your use case?

RAG can be genuinely transformative when the conditions are right. When they are not, it becomes an expensive way to learn that your data was never ready for production AI in the first place.

The honest answer to whether RAG suits your situation depends on your document quality, your query patterns, your latency requirements, and your team's capacity to own the retrieval layer over time. Those are not questions a vendor demo answers.

If you are at the point of evaluating a RAG build, a short scoping conversation can save months of misdirected effort. Better People's AI implementation advisory is designed for exactly this stage: working through the use case, the data landscape, and the build-or-buy trade-offs before any architecture decisions are locked in.

Is RAG the right approach for what you're trying to build?

href="/services/implementation" button="Talk to us about your use case" A 30-minute conversation covers your data environment, your retrieval requirements, and whether RAG, fine-tuning, or a simpler solution is the more honest fit.

Book a 30-minute discovery call →