RAG or Fine-Tuning: The Practical Test

RAG or fine-tuning: which approach delivers real value for enterprise AI? An extended decision framework with perspectives from Gartner, Forrester, IDC, KuppingerCole, Eckerson Group, and BARC, plus a pros-and-cons table for practitioners.

Using artificial intelligence is now a given for most enterprises. The real challenge starts only afterward: how does a language model get access to a company’s specific knowledge without costs, control, or compliance spinning out of hand? Two techniques dominate this debate: Retrieval Augmented Generation, or RAG, and the fine-tuning of language models. Progress Software, a vendor of AI-powered infrastructure and digital experience solutions, proposed a decision framework built around seven questions. It is worth extending that framework with practical experience, cost considerations, security questions, and the perspective of independent analyst firms, because seven questions alone rarely settle the decision in practice.

Two paths, one goal: getting knowledge into the model

RAG connects an existing language model to an external knowledge source at runtime, usually a vector database. A user query is first enriched with matching document excerpts before the model formulates an answer from them. The model’s own parameters remain unchanged; the company simply swaps out the knowledge base. Fine-tuning takes the opposite route: a pretrained model is further trained on curated, company-specific data sets, permanently changing its internal weights.

Both methods address the same underlying weakness of pretrained models, namely their static and mostly generic knowledge, but they take very different technical, organizational, and economic paths to get there. The approach traces back to a 2020 Meta AI research paper and has since evolved from an academic concept into a standard component of many AI architectures.

The seven-question compass in practice

Progress Software suggests making the decision based on seven simple questions. If company data changes frequently, for example through new products or policies, that favors RAG, because the system accesses current documents at runtime, whereas fine-tuning requires a costly and time-consuming new training cycle after every relevant change. If the model must be able to point to specific documents or data sources, RAG again has the edge, since it can back up answers with traceable sources, something a fine-tuned model generally cannot do reliably.

If, on the other hand, the priority is a consistent brand voice, a fixed writing style, or clearly defined response behavior, fine-tuning can provide more consistency, though many of these goals can, in practice, already be achieved with good system prompts and example context. If budget and time for recurring training cycles are lacking, the choice usually falls on RAG, because its iteration cycles are shorter and its upfront investment lower.

If verifiable, source-backed answers are business-critical, for instance in regulated industries, RAG offers a clear trust advantage. Fine-tuning, in turn, suits very narrow, largely static domains with specialized vocabulary, such as legal document drafting. And finally, successful fine-tuning already requires clean, structured, and labeled training data, whose preparation is frequently underestimated and can take weeks to months.

Thomas Schuller, Regional Director DACH at Progress Software, sums up the underlying observation: many companies initially assume a model must necessarily be trained on their own data. In practice, he notes, RAG approaches already cover the bulk of requirements, while fine-tuning is better suited as a complementary fine-adjustment for tone or a deeper understanding of the domain.

What the standard comparison often leaves out

The seven-question compass is a good starting point, but it leaves out three aspects that frequently determine success or failure in practice.

Cost over the full life cycle: Parameter-efficient methods such as LoRA and QLoRA, along with smaller language models in the seven-to-fourteen-billion-parameter range, have significantly lowered pure fine-tuning training costs over the past two years. But looking only at the training bill misses the fact that RAG generates its own ongoing costs: operating the vector database, refreshing embeddings, retrieval latency, and rising token costs with large context windows. A sound cost comparison must always consider the entire operating period, not just the first training run.

Security, governance, and the EU AI Act: RAG systems open up a new attack surface. Because answers are generated from retrieved documents, manipulated or injected content can lead to prompt injection or unintentionally expose sensitive data. This is exactly the category of threats addressed by the analyst firm KuppingerCole in its current market overview of GenAI protection solutions (see the next section). At the same time, a well-documented source chain in RAG systems can make it easier to meet EU AI Act evidentiary obligations, since answers can be traced back to the underlying document. Fine-tuning naturally does not offer this traceability, but it does reduce the attack surface created by an external knowledge base.

Data protection and data sovereignty: For companies in the DACH region, an additional question is where a RAG system’s vector database and embedding pipeline are actually operated. On-premises or EU-cloud solutions can become necessary as soon as personal or particularly sensitive company data flows into the knowledge base. This requirement affects RAG more than fine-tuning, because with RAG the raw data is kept permanently in a searchable form, whereas with fine-tuning it is only indirectly absorbed into the model weights.

For context on the hallucination problem, it is worth looking at RAG’s academic origin: the original 2020 paper by Lewis and colleagues at Meta AI already showed that connecting a model to external knowledge sources improves the reliability of generated answers, because the model no longer has to draw exclusively on its trained memory. How much hallucinations actually decrease in a given project, however, depends heavily on the quality of the knowledge base and the retrieval logic, and cannot seriously be condensed into a single, universally valid percentage. Anyone who needs solid figures of their own should determine them using an evaluation set built from their own domain documents.

How companies test the decision in practice

Regardless of which approach a company ultimately chooses, a structured test run before going live is advisable. A so-called golden test set has proven useful: a curated collection of real domain questions from the actual use case, for which the correct answer is already known. This set allows a RAG pipeline and a fine-tuned model to be compared under identical conditions, for example in terms of hit rate, response time, and the frequency of detectable false statements. It is important to measure not just answer quality but also retrieval quality in isolation, since a weak answer can stem from an unsuitable language model just as easily as from a poor set of matches from the vector database.

It is also worth running a simple red-team test, deliberately trying to trick the system into false statements using manipulated or contradictory documents. This testing practice aligns with KuppingerCole’s recommendation to continuously test GenAI systems against new attack patterns rather than relying on a one-time review before go-live. Only once both tests are satisfactory should a system move from a pilot project into productive DACH operation, where questions of data residency and DSGVO-compliant logging still need to be resolved.

A real interview study from the German-speaking region

How closely the framework outlined above matches actual practice is shown by a 2025 interview study conducted at the Department of Computer Science at the University of Innsbruck by Lorenz Brehme, Benedikt Dornauer, Thomas Ströhle, Maximilian Ehrhart, and Ruth Breu, presented at the KDIR 2025 conference. The researchers interviewed 13 practitioners from companies of varying size and industry who had implemented RAG systems themselves. According to the study’s acknowledgements, participating organizations included Vaadin, Akkodis, PPI AG, DeepOpinion, and other partners from the DACH region and Europe, with individual interview statements evaluated anonymously.

The results confirm the prioritization outlined in the analyst section: on a scale of 0 to 10, the practitioners surveyed rated data protection and privacy at an average of 8.9 points, the quality of generated answers at 8.7 points, and security at 8.5 points as the most important requirements, while ethical considerations and bias avoidance were prioritized noticeably lower, at an average of just 5.6 points. Twelve of the thirteen systems studied were still below Technology Readiness Level 7 at the time of the survey, meaning prototype or pilot stage rather than broad production use.

The practitioners consistently named data preparation as the biggest practical hurdle: unstructured formats such as PDFs, inconsistent chunking strategies, and correctly matching entities and abbreviations cost noticeably more effort in practice than building the RAG pipeline itself. One participant solved the problem of ambiguous identifiers by using a knowledge graph with ontologies instead of a classic vector database, to unambiguously link entities, synonyms, and abbreviations.

It was also notable that quality assurance in practice was carried out mostly manually, for example through test-question catalogs and thumbs-up/thumbs-down ratings from users, while automated, AI-supported evaluation methods from research have so far found little way into everyday company use. This observation matches the recommendation in the previous section to firmly anchor golden test sets and red-team tests in one’s own evaluation process.

What do the analysts say?

A look at independent market observers shows that RAG has established itself as the preferred starting point in broad enterprise practice, while fine-tuning remains an important but considerably smaller niche. One clarification matters here: the firms cited cover deliberately different specialties and are not interchangeable security analysts. Gartner and Forrester analyze IT topics across industries, KuppingerCole specializes in identity, access, and IT security, BARC and Eckerson Group come from business intelligence and data management, and Menlo Ventures, as explained below, is not an analyst firm at all but a venture-capital market observation.

Gartner: In its research document “Getting Started With Retrieval Augmented Generation”, Gartner describes RAG as a practical way to bring enterprise knowledge into language models without giving up control over proprietary data. The use of RAG is currently still a competitive differentiator, the analysts note, but is expected to become a fundamental competency for any organization using generative AI in the foreseeable future.

Forrester: In its two-part study “Forrester’s Guide To Retrieval-Augmented Generation”, the analyst firm frames RAG as a direct response to foundation models’ limitations in accuracy, relevance, and domain expertise. Forrester also describes how RAG is evolving from a pure retrieval building block into a more comprehensive platform with its own software ecosystem, paving the way toward agent-based automation.

IDC: In its FutureScape predictions for generative AI, IDC already expected that by 2025 two-thirds of enterprises would combine generative AI and RAG to enable domain-specific self-service knowledge management, and that this would improve decision quality in the affected processes by roughly half.

KuppingerCole: KuppingerCole, an analyst specializing in identity, access, and security, devotes a dedicated chapter to RAG systems in its Leadership Compass on GenAI protection solutions, because they open up attack surfaces such as prompt injection, indirect injection via manipulated documents, and data leakage. The analysts see especially strong demand for corresponding protective measures in Europe, particularly in Germany, driven by experience with the GDPR and the gradual implementation of the EU AI Act.

Eckerson Group: Eckerson Group, which focuses on data and analytics topics, advises data and AI leaders in its RAG research to structure the comparison between homegrown and commercial RAG solutions around a handful of targeted questions, for example around control over the data pipeline, maintenance effort, and long-term vendor lock-in, rather than being guided solely by marketing claims.

BARC: BARC, an analyst firm specializing in business intelligence and data management, found in a 2025 study published together with data-quality vendor Ataccama that 58 percent of surveyed organizations already have data observability programs in place, while 42 percent still do not fully trust the results of their AI models. As RAG adoption grows, BARC notes, unstructured data sources such as PDFs and long documents are moving further into the focus of data governance strategies, because they often bypass classic quality checks. Since this is a vendor-sponsored study, the figures should be read as a trend indicator rather than fully independent basic research.

Menlo Ventures: Not a classic analyst firm but a widely cited market observation from a venture capital firm, the annual survey by Menlo Ventures among US enterprise decision-makers is worth including. The 2024 edition shows a clear trend: the share of production AI architectures using RAG rose from 31 percent in 2023 to 51 percent in 2024, while fine-tuning was used in only 9 percent of production models. The follow-up edition, published in December 2025, qualitatively confirms this ranking again: prompt design remains the dominant technique, RAG follows in second place, while fine-tuning, together with tool calling and reinforcement learning, remains a niche used mainly by technically leading teams. Menlo Ventures did not, however, publish an equally precise percentage breakdown in its 2025 edition, so 31 versus 9 percent, and 51 versus 9 percent, remain the most precise publicly available comparison figures.

Pros and cons at a glance

The table below summarizes the key criteria and their practical implications.

Criterion	RAG	Fine-tuning
Data freshness	Very high, accessed at runtime	Low, requires a new training run
Traceability / sources	High, answers come with citations	Low, few source references
Consistent style / tone	Medium, steerable via prompting	High, permanently built into the model
Upfront effort and budget	Lower, faster iteration	Higher, even with LoRA/QLoRA
Ongoing costs	Retrieval, vector DB, token usage	Occasional retraining effort
Security risk	Prompt injection via documents	Smaller external attack surface
Data protection / sovereignty	Vector database needs governance	Knowledge embedded indirectly in weights
Best suited for	Dynamic domain knowledge, compliance	Stable vocabulary, fixed formats

Table 1: Criteria comparison of RAG versus fine-tuning for enterprise use. Editorial compilation based on the sources cited in the text.

Hybrid architectures are becoming the standard

The analyst views and the real-world figures paint a consistent picture: RAG is the more pragmatic entry point for most knowledge-intensive use cases, while fine-tuning plays to its strengths where behavior, format, or a very narrow domain vocabulary is the priority. In practice, the two approaches increasingly complement rather than exclude each other: a lightly fine-tuned model for tone and structure is combined with a RAG pipeline for current domain knowledge. This pattern delivers citable, on-brand, and simultaneously current answers, and Forrester describes it as the natural evolution of RAG toward agent-based automation.

Practical recommendations

For IT leaders facing this decision, four practical guardrails emerge from the sources cited here.

• Start with RAG when data changes frequently, source citations are required, or budget and time for recurring training cycles are lacking.
• Add fine-tuning selectively when tone, format, or a very stable domain vocabulary is the priority and sufficiently clean training data is available.
• Plan for EU AI Act and GDPR security requirements from the start, especially access controls on the vector database and protection against prompt injection.
• Calculate costs over the full life cycle, not just the first training run or the first integration of the knowledge base.

The value of this extended view lies in making the decision not purely on technical grounds but also on security, cost, and compliance grounds, since these factors more often determine project success in practice than the choice of model alone.

Questions and answers on RAG and fine-tuning