How to Fine-Tune an Open-Source LLM with LoRA and QLoRA

To fine-tune an open-source LLM with LoRA or QLoRA, you no longer need a cluster: a single GPU, a few hundred good examples and a well-chosen base model are often enough to change how a model behaves. The training script is the easy part. The decisions around it (whether to fine-tune at all, what data to use, which knobs matter, how to prove the result is better, how to serve it and whether the license lets you ship it) are where projects succeed or quietly fail. This guide walks through each of them.

When fine-tuning beats prompting or RAG, and when it does not

The most expensive fine-tuning mistake is training a model to solve a problem that was never a weights problem. Three techniques compete here, and each changes something different:

• Prompting changes how you ask. Instructions, few-shot examples and an output schema, with no training at all.
• Retrieval-augmented generation (RAG) changes what the model can see. It searches your documents at query time and puts the relevant passages in the prompt.
• Fine-tuning changes how the model behaves by default. It updates weights (or a small adapter) on your examples.

A useful test for any failure is a single question: is this a knowledge problem or a behavior problem? If the model does not know your product catalog, your policies or last week's prices, that is knowledge, and it belongs in retrieval. Fine-tuning facts into a model freezes them at training time and invites confident, outdated answers. If, instead, the model knows enough but will not reliably produce your JSON envelope, your clinical-note structure, your house tone or a narrow classification, that is behavior, and that is where fine-tuning earns its cost. The full case for retrieval, and its own failure modes, is in What Is RAG? A Practical Guide; building it to production standard is the subject of RAG: Retrieval-Augmented Generation in Practice.

Sequence them. Write the best prompt you can and score it on a fixed test set. If the remaining errors are knowledge-shaped, add retrieval and score again. Only if the residual errors are about consistent behavior, and the volume justifies the maintenance, gather examples and train an adapter. Production systems often end up with all three: a fine-tuned compact model for format and tone, retrieval for current facts, and a short system prompt for per-request guardrails. Working through that decision for real projects, then carrying it all the way to a self-hosted model, is the backbone of Fine-Tuning and Customizing Open-Source LLMs.

Why open weights, specifically

Managed fine-tuning from closed-model vendors is not a stable foundation to plan around. OpenAI, for example, notified developers on 7 May 2026 of changes to the availability of its self-serve fine-tuning: since 2 July 2026, organizations that have not run inference on a fine-tuned model in the previous 60 days can no longer create jobs, and from 6 January 2027 active customers will no longer be able to create new fine-tuning jobs either, while existing fine-tuned models keep serving until their base model is deprecated (see the OpenAI deprecations page). Fine-tuning an open-weight model gives you an artifact you control: you choose where it runs, where the training data lives, and when it changes.

How LoRA works, in one equation

Full fine-tuning updates every weight in the network, which means storing gradients and optimizer state for billions of parameters. Low-Rank Adaptation, introduced by Edward Hu and colleagues in LoRA: Low-Rank Adaptation of Large Language Models, takes a different route. It freezes the pre-trained weights and learns a small correction next to selected layers.

For a frozen weight matrix W of size d by k, LoRA adds two trainable matrices: A (r by k) and B (d by r), where the rank r is small, typically 8 to 64. The layer's output becomes:

h = W·x + (alpha / r) · B·A·x

Because r is tiny compared with d and k, the adapter has a small fraction of the parameters of the layer it modifies. In the PEFT library's default initialization, A is random and B starts at zero, so the adapter begins as an exact no-op and the model's behavior changes only as training moves B away from zero. The original paper reported that, compared with fully fine-tuning GPT-3 175B with Adam, LoRA reduced the number of trainable parameters by 10,000 times and GPU memory by 3 times, with no extra inference latency once the update is merged into the weights.

The practical consequences matter more than the math:

• Adapters are small files. You can keep many task-specific adapters for one base model.
• The base model is untouched. You can swap, compare or remove adapters without retraining.
• Merging is optional. W + (alpha/r)·B·A can be computed once and saved as a normal model, or applied on the fly at serving time.

QLoRA: LoRA on a 4-bit base model

LoRA shrinks what you train, but the frozen base model still has to sit in GPU memory. QLoRA, from Tim Dettmers and colleagues (QLoRA: Efficient Finetuning of Quantized LLMs), stores the frozen base in 4-bit precision and backpropagates through it into LoRA adapters kept in higher precision. The paper introduced three pieces that still define the technique:

• 4-bit NormalFloat (NF4), a data type designed for normally distributed weights.
• Double quantization, which quantizes the quantization constants themselves to save a bit more memory.
• Paged optimizers, which absorb memory spikes during training.

The headline result was fine-tuning a 65-billion-parameter model on a single 48 GB GPU. Two rules follow from how it works. First, 4-bit training only trains extra parameters: the quantized weights stay frozen, which is exactly what LoRA wants. Second, quantization is lossy, so QLoRA trades some fidelity in the frozen base for memory. For many adaptation tasks that trade is invisible in evaluation; for some it is not, which is one more reason to measure rather than assume.

Step 1: build the evaluation set before the training set

Write the test before the code. Collect real inputs from logs, tickets or domain experts, each with a known-good output, and freeze that set before you look at training data. It needs to cover the common cases, the edge cases that made you consider fine-tuning in the first place, and a handful of out-of-scope inputs where the right behavior is to refuse or hand off.

Score the base model with your best prompt on that set. This baseline is the number every training run must beat. Without it, a fine-tune that "feels better" in a few manual chats can be worse on the cases that matter, and nobody will know until users do.

Keep the evaluation set strictly separate from the training data. Deduplicate across the two, including near-duplicates that differ only in names or numbers, or your scores will measure memorization.

Step 2: prepare the data

Data quality decides the result far more than any hyperparameter. A few principles hold across tasks.

Quality and consistency over volume. The LIMA study from Meta researchers (Zhou et al., 2023) fine-tuned a 65B model on only 1,000 carefully curated prompt and response pairs and argued that most knowledge comes from pre-training, with fine-tuning mainly teaching format and style. For behavior-shaping tasks, a few hundred consistent, reviewed examples usually beat thousands of noisy ones. If two annotators would write different "correct" outputs for the same input, the model will learn the disagreement.

Use the format your trainer expects. Hugging Face TRL's SFTTrainer accepts conversational data (a messages list with roles) and prompt-completion data, and it applies the model's chat template automatically for conversational datasets. A conversational prompt-completion record looks like this:

{"prompt": [{"role": "system", "content": "You are the support triage assistant. Reply in the JSON envelope."},
            {"role": "user", "content": "My invoice shows two charges for October."}],
 "completion": [{"role": "assistant", "content": "{\"category\": \"billing_duplicate\", \"priority\": \"high\", \"reply\": \"...\"}"}]}

Train on the answer, not the question. For prompt-completion datasets, TRL computes the loss on the completion tokens only by default. For messages datasets, assistant_only_loss=True restricts the loss to assistant turns, but it requires a chat template with generation markers (TRL patches the template automatically for some known model families; check yours). Training on the prompt tokens as well wastes capacity on reproducing user text.

Match the chat template of the model you will serve. The template (role markers, special tokens, end-of-turn token) used in training must be the one used at inference. A mismatch is one of the most common causes of a fine-tune that rambles or never stops.

Clean and protect. Remove duplicates, broken records and outputs you would not want repeated. Strip or pseudonymize personal data unless you have a lawful basis to train on it, and record where every source came from. Check the terms of any model you use to generate synthetic examples: some licenses restrict using outputs to train other models.

Split three ways. Training, validation (for watching loss during training) and the frozen evaluation set from step 1.

Step 3: the LoRA and QLoRA hyperparameters that matter

LoRA has few knobs, and most of the defaults you will see copied around are reasonable starting points rather than laws. These are the ones that change outcomes:

A few of these deserve more than a table row.

Target all linear layers. In PEFT, target_modules="all-linear" puts adapters on every linear layer except the output head, which is also how QLoRA was designed. Thinking Machines' study LoRA Without Regret (September 2025) found that attention-only LoRA underperformed even when the number of trainable parameters was matched, and that adding attention adapters on top of MLP adapters brought no additional benefit.

The learning rate is higher than you expect. The same study found the optimal learning rate for LoRA to be about 10 times that of full fine-tuning, and TRL's documentation suggests around 1e-4 when training adapters. Note that SFTConfig defaults to 2e-5, so set it explicitly. A short sweep (for example 5e-5, 1e-4, 2e-4) on the validation set is cheap and often the single best use of your compute.

Watch the batch size. The Thinking Machines results also showed LoRA to be less tolerant of large batch sizes than full fine-tuning, and that raising the rank does not fix it. If you increase gradient accumulation to fit a bigger effective batch, check that quality does not drop.

Rank is capacity, alpha is strength. Because the update is scaled by alpha / r, raising r while keeping alpha fixed weakens the update. PEFT's use_rslora=True switches the scaling to alpha / sqrt(r), which is worth trying at high ranks; use_dora=True (DoRA) separates magnitude and direction and can help at low ranks, at some extra training cost.

Know when LoRA runs out of room. The same study found that for typical post-training datasets LoRA has enough capacity to match full fine-tuning, and that it falls behind when a dataset carries more information than the adapter can hold. The paper LoRA Learns Less and Forgets Less (Biderman et al., 2024) is the other side of the coin: on large code and math datasets LoRA underperformed full fine-tuning, but it preserved the base model's abilities outside the target domain better. For behavior shaping, that trade is usually the one you want.

A minimal QLoRA training script

The script below uses the current Hugging Face stack: TRL's SFTTrainer with a PEFT LoraConfig and a bitsandbytes 4-bit configuration, which together give QLoRA. Replace the model ID with an open-weight model whose license you have checked (more on that below).

import torch
from datasets import load_dataset
from peft import LoraConfig
from transformers import BitsAndBytesConfig
from trl import SFTConfig, SFTTrainer

BASE_MODEL = "your-org/your-open-weight-model"  # license checked, chat template known

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",          # NF4, from the QLoRA paper
    bnb_4bit_use_double_quant=True,     # quantize the quantization constants
    bnb_4bit_compute_dtype=torch.bfloat16,
)

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules="all-linear",
    task_type="CAUSAL_LM",
)

data = load_dataset("json", data_files={"train": "train.jsonl", "validation": "val.jsonl"})

args = SFTConfig(
    output_dir="triage-qlora",
    model_init_kwargs={"dtype": torch.bfloat16},
    learning_rate=1e-4,
    num_train_epochs=2,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    max_length=2048,
    eval_strategy="steps",
    eval_steps=50,
    logging_steps=10,
)

trainer = SFTTrainer(
    model=BASE_MODEL,
    args=args,
    train_dataset=data["train"],
    eval_dataset=data["validation"],
    peft_config=lora_config,
    quantization_config=bnb_config,   # 4-bit base + LoRA = QLoRA
)
trainer.train()
trainer.save_model()  # saves the adapter, not a full model copy

Gradient checkpointing is on by default in SFTConfig, which trades compute for memory. Watch two curves: training loss should fall smoothly, and validation loss should fall and then flatten. When validation loss starts rising while training loss keeps dropping, the adapter is memorizing; stop earlier, lower the rank or learning rate, or add data. For faster single-GPU runs, Unsloth and Axolotl wrap the same ideas with their own optimizations and configuration styles; the concepts above carry over unchanged.

Step 4: evaluate before and after

Training loss tells you the model fits the data. It does not tell you the model is better at the job. Run the frozen evaluation set from step 1 on three systems: the base model with your best prompt, the fine-tuned model with the short prompt you plan to ship, and, if you use retrieval, both with the same retrieved context. Compare them on the metrics that define success for your task:

• Task metrics. Accuracy or F1 for classification, exact or schema-valid output rate for structured responses, rubric scores for open-ended text.
• Hard checks in code. JSON parses, required fields are present, values come from allowed lists. These catch regressions that a judge model might forgive.
• Regression on general ability. Fine-tuning can erode skills you did not train on. Keep a small set of general prompts (instruction following, reasoning, refusals for unsafe requests) and compare before and after. EleutherAI's lm-evaluation-harness can load a PEFT adapter directly with --model_args pretrained=<base>,peft=<adapter> if you also want standard benchmarks.
• Human review of disagreements. Read every case where the base and the fine-tune disagree. If you use an LLM as a judge, calibrate it against human labels first.

Re-run the same suite after every change: new data, new rank, a merged or quantized export. The methods for building that suite, from test sets to judges to regression gates in CI, are covered in LLM Evaluation and Testing: Shipping Reliable AI, and the short version is in LLM Evaluation: How to Test AI Applications.

Step 5: serve the adapter

You have a small adapter and a base model. There are two ways to ship them.

Keep the adapter separate when you have several variants (per customer, per task, per language) over the same base. vLLM can load one base model and apply different LoRA adapters per request (vLLM LoRA documentation):

vllm serve your-org/your-open-weight-model \
    --enable-lora \
    --lora-modules triage=./triage-qlora \
    --max-lora-rank 16

Clients then select the adapter through the model field of an OpenAI-compatible request ("model": "triage"). vLLM can also load adapters at runtime through an endpoint, but its documentation warns that this carries security risks and should not be enabled in production unless the environment is isolated and fully trusted.

Merge the adapter when you need a single standalone model, for example to quantize it for local serving. PEFT's merge_and_unload() folds the adapter into the base weights and returns a plain model (it is not an in-place operation, so assign the result). For a QLoRA adapter, load the base model in 16-bit precision for the merge rather than the 4-bit training copy: PEFT documents that merging into quantized weights dequantizes, adds the adapter and quantizes again, so rounding error can accumulate. Merge first, evaluate the merged model, and only then quantize for inference (for example to GGUF with the conversion script in llama.cpp, which Ollama can import with a FROM line in a Modelfile). Re-run your evaluation after each of those steps; every quantization is another lossy transformation.

Two habits save pain later. Always confirm that the adapter is applied to exactly the base checkpoint it was trained on; PEFT records it in adapter_config.json. And keep the adapter plus its training configuration even after you merge: it is the source you can rebuild from when the base model gets a patched release.

Licensing: open weights is not the same as open source

Downloadable weights tell you that you can get the model, not what you may do with it. Before you fine-tune anything you plan to ship, read the license on the official model card and in the repository, not a summary. A few current examples show how much it varies, even within one vendor:

The Llama 4 Community License shows the clauses that most often catch teams out. Entities with more than 700 million monthly active users in the month before the release date must request a separate license from Meta. If you distribute a derivative model, its name must begin with "Llama", and "Built with Llama" must be displayed prominently on a related website, interface or documentation. And the Llama 4 Acceptable Use Policy states that, for the multimodal Llama 4 models, the rights in Section 1(a) of the license are not granted to individuals domiciled in the EU or companies whose principal place of business is in the EU (end users of a product built on them are not affected). For a company based in the EU, that alone can rule a base model out.

Check three licenses, not one: the base weights, the training data (including the terms of any model used to generate synthetic data) and the tools. Write the result down next to the adapter. A step-by-step way to read a model license before training, alongside the rest of the pipeline, is part of Fine-Tuning and Customizing Open-Source LLMs. This section is general information, not legal advice; for a commercial deployment, have counsel read the actual license text.

A fine-tuning checklist

• A strong prompt (and retrieval, if the gap is knowledge) was measured first, and the remaining gap is behavior.
• A frozen evaluation set exists, deduplicated against the training data, with a baseline score.
• Training examples are consistent, reviewed, stripped of personal data you have no lawful basis to use, and in the trainer's expected format.
• The chat template in training matches the one used in serving.
• Loss is computed on completions or assistant turns only.
• The learning rate was set explicitly and swept; validation loss was monitored.
• Task metrics, hard format checks and a general-ability regression set were compared before and after.
• The adapter is served on its exact base checkpoint, or merged into a 16-bit base and re-evaluated after every quantization.
• Base model, data and tool licenses are recorded next to the artifact.

Frequently Asked Questions (FAQ)

What is the difference between LoRA and QLoRA? LoRA freezes the base model and trains small low-rank adapter matrices next to selected layers. QLoRA does the same, but stores the frozen base model in 4-bit precision (NF4) so it fits in far less GPU memory. The adapters are trained the same way; QLoRA trades some precision in the frozen base for memory.

How many examples do I need to fine-tune an LLM? For behavior such as format, tone or a narrow classification, a few hundred consistent, reviewed examples are often enough to see a clear effect, and quality matters more than volume. Teaching broad new skills needs much more data, and teaching changing facts is better done with retrieval.

What LoRA rank and learning rate should I start with? A common starting point is rank 16 with alpha 32, adapters on all linear layers, and a learning rate around 1e-4. Then sweep the learning rate and adjust the rank based on validation loss and your evaluation set, not on someone else's defaults.

Should I fine-tune or use RAG? Use RAG when the model lacks knowledge, especially private or frequently changing facts, or when answers must cite sources. Fine-tune when the model knows enough but behaves inconsistently: format, tone, structure or a narrow task. Many production systems combine both.

Can I use a fine-tuned open-source model commercially? Often yes, but it depends on the license of the base model, the data and any model used to generate training data. Apache 2.0 models are permissive; custom licenses such as the Llama 4 Community License add conditions on naming, attribution, scale and, for some models, EU use. Read the official license text before you ship.

Conclusion

Fine-tuning an open-source LLM with LoRA or QLoRA is now a single-GPU job, but it is the right job only when the gap is behavior rather than knowledge. Measure a strong prompt first and add retrieval for facts. Then build a frozen evaluation set, curate a few hundred consistent examples, train adapters on all linear layers with an explicitly tuned learning rate, and compare base and fine-tune on task metrics, hard checks and general ability. Serve adapters separately when you need many variants, merge into a 16-bit base when you need one standalone model, and re-evaluate after every export. Read the license before the first training run, not after the launch.