Last year, at SAP TechEd in Berlin, I presented one of my favorite topics, 😉 bringing a custom fine-tuned Hugging Face model into the SAP AI landscape through the Bring Your Own Model (BYOM) concept with SAP AI Core. see my Blog
Since then, one question has continued to surface: Is fine-tuning still relevant?
Over the past two years, you have probably seen headlines such as “Fine-Tuning LLMs Is Dead,” “Stop Fine-Tuning Your LLMs,” “Why Fine-Tuning Is a Waste of Time,” or “RAG Replaced Fine-Tuning.”
These statements attract attention and start discussions. However, they often reduce a complex architectural decision to a simple either-or debate. But they do not tell the full story. Fine-tuning is not dead. It is also not the answer to every generative AI problem. The real question is not whether we should choose fine-tuning or Retrieval Augmented Generation (RAG). The better question is when each technique creates value, and how they can work together in enterprise scenarios.
For SAP developers and architects, this distinction matters. A model may need access to current business documents through RAG, while also needing to learn how to classify tickets, generate predictable structures, follow SAP-specific conventions, or produce output that fits reliably into an enterprise workflow.
The truth is actually more nuanced; fine-tuning has not disappeared. It has become more specialized. Prompting and RAG now handle many use cases effectively. Fine-tuning remains relevant for specific, high-value scenarios where consistency, structure, domain adaptation and efficient execution matter most.
Let us look at why fine-tuning is still very much alive. 🚀
What Is Fine-Tuning?
Before diving into the debate, let us clarify what fine-tuning actually means, especially for readers coming from ABAP, SAP BTP or enterprise integration backgrounds.
Fine-tuning is the process of taking a pretrained Large Language Model (LLM) and continuing its training on a curated dataset so that it becomes better at a specific task, domain or expected behavior.
A foundation model already has broad language capabilities. Fine-tuning helps it become more reliable for a particular purpose.
For example, an LLM may already understand what a support ticket is. Through fine-tuning, it can learn how your organization classifies tickets, which component codes it uses, and how the expected output must be formatted.
How Fine Tuning Works Technically
At a simplified level, the process usually looks like this:
| 1. Select a base model | Start with an existing model, such as an open-source model from Hugging Face or another supported model family |
| 2. Prepare training data | Create high-quality examples that represent the target task and desired output |
| 3. Train the model | Adjust model parameters or selected adapter parameters, based on the examples |
| 4. Evaluate results | Test quality, consistency, safety, and business relevance |
| 5. Deploy and monitor | Make the adapted model available for applications and track its behavior in operation |
Traditional full fine-tuning updates all model parameters. Today, many teams prefer parameter-efficient approaches that require less compute and memory.
Types of Fine Tuning
| Full Fine Tuning | Updates all model parameters | When maximum adaptation is required and sufficient compute, data and governance are available |
| LoRA, Low Rank Adaptation | Freezes the base model and trains small additional parameter matrices | A practical choice for adapting large models with lower cost and memory requirements |
| QLoRA | Combines quantization with LoRA to reduce memory requirements further | When training resources are limited or larger open source models must be adapted efficiently |
| Supervised Fine Tuning | Trains the model on input and expected output examples | Classification, extraction, transformation and controlled response patterns |
| Instruction Tuning | Teaches the model to follow instructions and expected response styles | Assistants, workflow completion and enterprise task automation |
| Retrieval-Aware Fine-Tuning | Trains the model to use retrieved documents more effectively | Domain-specific RAG scenarios where the model must distinguish relevant evidence from noise |
LoRA is one of the most important developments in modern fine-tuning. Instead of updating all parameters in a large model, it freezes the original model weights and trains smaller low-rank matrices. Hu et al. showed that this can substantially reduce trainable parameters and memory requirements while achieving competitive results.
QLoRA extended this concept by combining LoRA with quantization. Dettmers et al. demonstrated that a large model could be fine-tuned with much lower memory requirements while preserving strong performance.
This is one reason why fine-tuning has not disappeared. It has become much more practical.
Why Fine Tuning Still Matters
The argument for fine-tuning should not rely only on personal experience or technology trends. Research provides strong reasons why model adaptation remains relevant.
1. Parameter-Efficient Fine-Tuning Made Adaptation More Practical
The LoRA paper by Hu et al. introduced an efficient way to adapt large language models by training a small number of additional parameters rather than updating the full model.
This is important for enterprise scenarios because it reduces the cost and complexity of adaptation. Instead of treating fine-tuning as an expensive project available only to companies with large GPU environments, LoRA makes targeted adaptation more achievable.
QLoRA, introduced by Dettmers et al., took this further by reducing memory requirements through quantization. Their work showed that large models can be adapted efficiently on more limited hardware while maintaining strong results.
which means that for some cases, exploring custom open-source models, these techniques make fine-tuning a realistic architectural option rather than a purely theoretical one.
2. RAG Solves a Different Problem Than Fine-Tuning
RAG is highly valuable when a model needs access to current, private, or frequently changing information.
For example, an assistant answering questions based on SAP Help documentation, SAP Notes, internal process manuals or project documents should usually retrieve this information at runtime rather than attempt to memorize it during fine tuning.
Lewis et al. introduced RAG as a way to combine pretrained language models with retrieved external knowledge. This enables models to answer knowledge-intensive questions using relevant documents.
However, RAG does not automatically teach a model how your company expects a response to be classified, structured or formatted.
| Current business documents and changing knowledge | RAG |
| Consistent output schema | Fine-tuning or structured output controls |
| Company-specific task behavior | Fine tuning |
| Fast prototype | Prompting |
| Knowledge plus specialized behavior | RAG combined with fine-tuning |
In other words, RAG helps a model access the right information. Fine-tuning can help a model behave in the right way.
3. Fine-Tuning and RAG Can Be Combined
The debate often presents RAG and fine-tuning as competing approaches. Recent research shows that they can complement each other. The RAFT paper by Zhang et al. studied Retrieval Augmented Fine Tuning for domain-specific RAG scenarios. The approach trains a model not only to answer questions from relevant documents but also to handle distractor documents and use retrieved context more effectively. This is highly relevant for enterprise environments. Real-world retrieval systems may return documents that are partially relevant, outdated, or similar but not correct for the current case. The goal is not always to choose between RAG and fine-tuning. In some cases, the strongest architecture may combine both.
4. Fine-Tuning Helps Models Follow Expected Behavior
Instruction tuning research also shows that fine-tuning plays an important role in teaching models how to respond.Ouyang et al. showed that models trained using demonstrations and human feedback could follow user intent more effectively than a much larger base model. The broader lesson is important: a larger general model is not automatically better for every specific enterprise task. A smaller or adapted model can perform better when it has been trained for the required behavior.
Structured Outputs Are Where Fine-Tuning Can Shine
Enterprise AI applications often require more than a conversational answer.A business application may require a predictable structure such as JSON, an extraction schema, a ticket category, a mapping result, or a workflow decision.
For SAP developers, examples include:
| Consistent structured extraction | Extracting purchase order information into a defined schema |
| Stable classification | Assigning support tickets to SAP components |
| Controlled code patterns | Generating ABAP or CDS examples aligned with internal development standards |
| Consistent mapping proposals | Suggesting IDoc, EDI, or master data mappings |
| Workflow compliant responses | Returning outputs that can be processed by SAP Build Process Automation or custom applications |
Prompting can be sufficient for a prototype. RAG can supply missing information. However, for frequent and repeatable tasks, fine-tuning can help make the behavior more stable and task-specific.
This does not mean that fine-tuning removes the need for validation. Production systems still require schema checks, evaluation, monitoring, and human review where business risk is high.
Domain Adaptation Still Matters
SAP environments include specialized language, business processes and technical artifacts. Terms such as CDS View, BAdI, RAP, IDoc, OData service, contract account, clearing status or insurance object are not only words. They often represent specific structures, business rules and implementation patterns. A model designed for general conversation may understand parts of this language. A model adapted on carefully prepared examples can become more consistent in using it for a defined task.
Fine-tuning may be useful when a model must:
| Use SAP-specific terminology correctly | Differentiate a CDS View from an OData service or RAP business object |
| Follow internal development standards | Generate code patterns aligned with naming and design conventions |
| Produce task-specific outputs | Return a support classification, mapping proposal or structured extraction result |
| Support a particular industry process | Work with insurance, finance, procurement or settlement terminology |
Again, this should not be confused with loading current SAP documentation into a model. For current knowledge, RAG is usually the better option. Fine-tuning is more suitable for teaching behavior, format, and specialized task patterns.
When Fine-Tuning Is the Right Tool
Fine-tuning is not required for every use case. But it can be highly valuable in the following scenarios:
| Structured output generation | Encourages predictable schema-compliant responses | Returning JSON structures for downstream services |
| Domain-specific task behavior | Teaches terminology and expected output patterns | SAP support ticket classification |
| Code generation aligned with standards | Learns company conventions and recurring design patterns | ABAP, CDS or SAPUI5 code assistance |
| Repetitive high-volume tasks | Can improve task consistency at scale | Mass classification or document extraction |
| Smaller model deployment | Adapts efficient models for a narrow task | A specialized assistant exposed through SAP AI Core |
| Retrieval-based domain assistants | Helps the model use retrieved evidence more effectively | Internal SAP process documentation assistant using RAG and fine-tuning |
When Fine-Tuning Is Probably Not the First Choice
Fine-tuning should not be the default reaction to every AI requirement.
| Access to new SAP documentation or changing policies | RAG |
| Quick proof of concept | Prompting with an available model |
| General-purpose summarization or drafting | Base model with prompt design |
| Unclear business value | Evaluation and prototyping before training |
| Limited or poor quality training data | Improve data and evaluation before fine-tuning |
A good enterprise AI strategy does not start by training a model. It starts by defining the problem, the required quality, the available data, the integration constraints, and the evaluation criteria.
Conclusion
Fine-tuning LLMs is not dead. It has evolved from a general solution into a specialized tool for specific, high-value scenarios, especially in enterprise environments.
For SAP developers and architects, fine-tuning remains relevant when a model must produce consistent structured outputs, learn SAP-specific task behavior, support repeatable workflows, or make smaller models useful for targeted applications.
At the same time, RAG is often the better choice for current knowledge, internal documents and frequently changing business context.
The most successful AI architectures will not choose between fine-tuning and RAG as competing ideologies. They will combine prompting, retrieval, adaptation, evaluation and enterprise integration according to the actual business requirement.
So the next time you see the statement “fine tuning is dead,” remember:
Fine-tuning did not disappear. It became more focused, more efficient, and, for the right SAP use case, still highly valuable.
What is your experience with fine-tuning? Have you seen it deliver clear value, or do you find prompting and RAG sufficient for most enterprise scenarios? I would be happy to hear your thoughts in the comments. 💬
References
Hu, E. J., Shen, Y., Wallis, P., Allen Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2022). LoRA: Low Rank Adaptation of Large Language Models. International Conference on Learning Representations.
https://arxiv.org/abs/2106.09685
Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. Advances in Neural Information Processing Systems, 36.
https://arxiv.org/abs/2305.14314
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S., & Kiela, D. (2020). Retrieval Augmented Generation for Knowledge Intensive NLP Tasks. Advances in Neural Information Processing Systems, 33.
https://arxiv.org/abs/2005.11401
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., & Lowe, R. (2022). Training Language Models to Follow Instructions with Human Feedback. Advances in Neural Information Processing Systems, 35.
https://arxiv.org/abs/2203.02155
Zhang, T., Patil, S. G., Jain, N., Shen, S., Zaharia, M., Stoica, I., & Gonzalez, J. E. (2024). RAFT: Adapting Language Model to Domain Specific RAG. Conference on Language Modeling.
Bluesky
