Fine-tuning

During fine-tuning, this model is further adapted with a comparatively small, curated data set so that it performs a specific task better, hits a desired tone of voice or masters the technical vocabulary of an industry. In other words, you do not build from scratch but refine an existing foundation.

How fine-tuning works

During fine-tuning, the internal parameters (weights) of the model are adjusted on the basis of the new example data. Typically, the training data set consists of pairs of input and desired output, such as customer enquiries and ideal answers. The model learns from these examples to shift its behaviour in the desired direction. Because fully retraining all the billions of parameters is very compute-intensive, resource-saving methods have become established, above all LoRA (low-rank adaptation) and related methods of "parameter-efficient fine-tuning". Here, only a small number of additional parameters are trained while the base model remains largely frozen, which reduces costs and computing requirements considerably.

Distinction: embeddings

Embeddings are something fundamentally different from fine-tuning and are often confused with it. An embedding is a numerical representation, a vector of many numbers, that maps the meaning of a text, image or product in a mathematical space. Content with a similar meaning lies close together in this space. Embeddings do not change the model; they are the tool for making similarity and meaning measurable. They are used, for example, for semantic search, recommendations and, particularly importantly, for retrieval augmented generation (RAG), in which a model retrieves relevant documents at runtime and incorporates them into its answer without being retrained.

Fine-tuning, RAG or prompting: when to use what?

One of the most common practical questions is: should you fine-tune a model, build a RAG system or simply instruct it cleverly (prompting)? The three approaches solve different problems:

An important rule of thumb: fine-tuning is good at teaching the model a particular behaviour or style, but poor at supplying it with current factual knowledge. Anyone who wants to "teach" a model their own changing product catalogue is almost always better served by RAG: there, the facts are kept in a searchable knowledge base and can be updated without retraining the model. Fine-tuning is worthwhile when a consistent style, a fixed answer format or a specialised behaviour is required that cannot be achieved reliably via a prompt.

A concrete example

A retailer wants an AI customer service that, first, answers in the brand's own friendly, concise tone and, second, always knows the current range and delivery times. The right solution combines both worlds: the tone of voice and answer format are anchored via fine-tuning (or often already via good prompting); the current product and delivery knowledge comes via RAG from a constantly updated knowledge base. If you tried to "burn" the range into the model via fine-tuning, it would already be outdated at the next change of range, a classic and expensive mistake.

Costs, risks and limits

Fine-tuning does not run by itself. It needs a high-quality, representative training data set; poor data leads to poor behaviour. There is a risk of "catastrophic forgetting", in which the model loses general capabilities as a result of the further training. In addition, there are costs for training, versioning and fine-tuning again when requirements change. And a fine-tuned model has to be maintained: if a better base model appears, the work may have to be done again. For these reasons, the pragmatic order is: prompting first, then RAG, and fine-tuning only if neither is sufficient.

Common misconceptions

First: fine-tuning is not the standard way of giving a model "your own data"; RAG is usually more suitable for that. Second: fine-tuning does not make a model generally "more intelligent" but more specialised; it can become better in one place and worse in another. Third, embeddings and fine-tuning are often lumped together even though they do completely different things: embeddings measure meaning, fine-tuning changes the model. Fourth: more training data is not automatically better; quality and fit beat quantity.

Outlook

With ever more capable base models and larger context windows, the boundary is shifting: much of what used to require fine-tuning can now be solved with good prompting or RAG. At the same time, efficient methods such as LoRA are making fine-tuning cheaper and more accessible. For most companies, the recommendation remains to start with the lighter tools and to use fine-tuning specifically where consistent, specialised behaviour brings real added value. The Wikipedia article on fine-tuning in deep learning offers a technical classification of the term. We clarify which level makes sense for your use case in our AI consulting.

FAQ

What is fine-tuning in simple terms?
The further training of a finished AI model with your own specialised example data so that it masters a specific task or style better.

What is the difference between fine-tuning and embeddings?
Fine-tuning changes the model itself. Embeddings are numerical representations of meaning that do not change the model but make similarity measurable, for example for search and RAG.

Should I use fine-tuning or RAG?
For current factual knowledge (e.g. your own product catalogue), usually RAG. For a consistent style, tone of voice or fixed behaviour, fine-tuning. A combination is often ideal. An AI agency can clarify which variant suits your data situation and budget as part of a short feasibility check.

What is LoRA?
LoRA (low-rank adaptation) is a resource-saving fine-tuning method that trains only a few additional parameters and leaves the base model largely unchanged, which saves computing power and costs.