LoRA fine-tuning for LLMs on a cloud GPU

LoRA fine-tuning lets you adapt a large language model without retraining all of its parameters.

Instead of changing billions of weights across the base model, LoRA adds small trainable matrices to selected layers and keeps the original weights frozen. You train those additional parameters on your own examples, then save them as an adapter that can be loaded alongside the original model.

For an 8B model, this changes the hardware problem considerably. Full fine-tuning can require far more GPU memory than inference because training has to keep gradients, optimizer states, activations, and other intermediate data in memory. LoRA reduces that burden by training only a fraction of the model's parameters.

QLoRA goes further. It loads the frozen base model in 4-bit precision and trains LoRA adapters on top of it. That makes fine-tuning useful models on a single GPU much more practical. Hugging Face's current PEFT and TRL libraries support this workflow directly.

In this guide, we'll fine-tune Llama 3.1 8B Instruct with QLoRA on one RTX 5090.

The setup uses:

• 1 × NVIDIA RTX 5090 with 32GB VRAM
• Llama 3.1 8B Instruct
• 4-bit NF4 quantization
• LoRA adapters
• Hugging Face Transformers, PEFT, TRL, and bitsandbytes
• a conversational JSON dataset

If you want the infrastructure without the walkthrough, see training and fine-tuning with Hivenet.

What is LoRA fine-tuning?

LoRA stands for Low-Rank Adaptation.

The method starts from an observation about fine-tuning large neural networks: you often do not need to update the full weight matrix to teach a pretrained model a narrower task.

LoRA leaves the original matrix frozen and learns its update through two much smaller matrices.

You do not need the linear algebra to use it, but the difference matters.

Suppose a layer contains a large weight matrix:

W

Full fine-tuning changes that entire matrix.

LoRA instead learns an update:

W + ΔW

where ΔW is represented by the product of two low-rank matrices.

Because the rank is much smaller than the original matrix dimensions, the number of trainable parameters falls sharply.

The base model remains intact. Your fine-tuned behavior lives in the adapter.

That has several practical consequences:

• training needs less GPU memory
• adapters take less storage than a second complete model
• one base model can be paired with different adapters
• experiments are easier to discard or replace
• you can preserve the original model instead of producing a new full checkpoint after every run

LoRA was introduced specifically as a parameter-efficient alternative to full model adaptation. citeturn780885academia24

LoRA, QLoRA, and full fine-tuning are different

These terms are often mixed together, which makes hardware advice confusing.

Plain LoRA reduces the amount of model state you train.

‍

‍

QLoRA also reduces the memory occupied by the frozen base model.

The original QLoRA work used a 4-bit quantized model while preserving trainable LoRA parameters at higher precision. Current Hugging Face tooling exposes the same basic pattern through BitsAndBytesConfig and PEFT.

For this tutorial, QLoRA is the better default. There is little reason to occupy most of a 32GB GPU with an 8B model at higher precision when the purpose of the exercise is parameter-efficient fine-tuning.

What is LoRA good for?

Fine-tuning works best when you want to change how a model behaves.

Good examples include:

• teaching a consistent output format
• adapting terminology for a specialist domain
• learning a classification or extraction task
• following a particular instruction pattern
• producing structured responses
• improving performance on a narrow type of problem
• adapting style or tone
• learning how examples from a particular workflow should be handled

It is much less convincing as a way to keep a model informed about facts that change every week.

If your main requirement is "answer questions from these documents," changing the model weights may solve the wrong problem. A retrieval system lets you change the underlying documents without retraining the model.

That is the distinction behind RAG with Hivenet.

Fine-tune when you need the model to behave differently. Use retrieval when you mainly need it to know different information at request time.

Real applications often use both.

How much GPU memory does LoRA fine-tuning need?

There is no fixed VRAM requirement for "LoRA."

Memory depends on:

• model size
• base-model precision
• sequence length
• batch size
• LoRA rank
• which layers receive adapters
• optimizer
• gradient checkpointing
• training framework

This is why statements such as "LoRA needs 16GB" are not useful on their own.

For the Llama 3.1 8B example here, QLoRA gives us considerable room on an RTX 5090 because the base model is loaded at 4-bit rather than BF16.

The RTX 5090 has 32GB of VRAM. Our RTX 5090 VRAM guide explains the weights-only calculation and why training memory is different from inference memory.

As model size increases, the calculation changes. Fine-tuning a 70B model is a different infrastructure problem from adapting an 8B model. We cover that larger memory boundary in our planned Llama 3.3 70B GPU requirements guide.

Why use Llama 3.1 8B for this tutorial?

Llama 3.1 8B is large enough to demonstrate a realistic LLM fine-tuning workflow without turning the tutorial into a multi-GPU exercise.

Meta provides pretrained and instruction-tuned Llama 3.1 variants at 8B, 70B, and 405B parameters. The 8B Instruct model supports a 128K context window, although using that full context during training would require far more memory than the modest sequence length we use here.

We will use:

meta-llama/Llama-3.1-8B-Instruct

You must accept Meta's Llama 3.1 license conditions through Hugging Face before downloading the model. The repository is access-controlled and subject to the Llama 3.1 Community License.

Do that before launching an expensive GPU instance. There is no reason to pay for a GPU while waiting for model access.

Step 1: launch one RTX 5090

Create a GPU instance in Compute with Hivenet.

For this tutorial, choose:

1. 1 × RTX 5090
2. a PyTorch environment or suitable Ubuntu setup
3. enough disk space for the base model, datasets, checkpoints, and outputs
4. SSH or Jupyter access

Hivenet supports PyTorch images and GPU instances for training workloads. The RTX 5090 has 32GB of VRAM, and Compute bills GPU use by the second.

If this is your first instance, follow the Compute quickstart.

Connect to it and confirm the GPU is visible:

nvidia-smi

Then check PyTorch:

python - <<'PY'
import torch

print("PyTorch:", torch.__version__)
print("CUDA available:", torch.cuda.is_available())

if torch.cuda.is_available():
   print("GPU:", torch.cuda.get_device_name(0))
PY

Do this before setting up the training stack.

Step 2: install the fine-tuning libraries

Create an isolated Python environment if your template does not already provide one:

python -m venv ~/lora-env
source ~/lora-env/bin/activate

Upgrade pip:

pip install --upgrade pip

Then install the Hugging Face training stack:

pip install --upgrade \
 transformers \
 datasets \
 accelerate \
 peft \
 trl \
 bitsandbytes \
 huggingface_hub

TRL integrates directly with PEFT for LoRA and QLoRA training. QLoRA also requires bitsandbytes for 4-bit loading.

Step 3: authenticate with Hugging Face

After you have accepted access to the Llama repository, authenticate from the instance:

hf auth login

Paste a Hugging Face user access token when prompted.

You can confirm the active account with:

hf auth whoami

The current Hugging Face CLI uses hf auth login for persistent local authentication.

Treat the token as a credential. Do not put it directly into a training script or commit it to a repository.

Step 4: prepare your training data

The model will learn from the examples you give it, so data quality matters more than people often admit.

For conversational supervised fine-tuning, TRL accepts datasets containing structured messages with roles and content.

Create a file called:

train.jsonl

Each line should contain one training example:

{"messages":[
 {"role":"user","content":"Rewrite this incident report as a concise technical summary: The service became unavailable after the database connection pool was exhausted."},
 {"role":"assistant","content":"The service became unavailable after exhausting its database connection pool."}
]}
{"messages":[
 {"role":"user","content":"Rewrite this incident report as a concise technical summary: Requests slowed after a deployment increased memory use across all workers."},
 {"role":"assistant","content":"Requests slowed after a deployment increased memory consumption across the worker pool."}
]}

Those examples are deliberately simple.

A real training dataset should contain enough variation to teach the behavior rather than reward memorization of a handful of phrases.

Your data should also resemble the requests the model will receive after deployment.

If production prompts contain long documents but your training examples contain one-sentence inputs, you have trained on a different problem.

Do not start by dumping every document you own into the dataset

More examples can help.

More noise does not.

Before training, inspect the dataset for:

• duplicated examples
• contradictory answers
• accidental secrets
• personally identifiable information you do not need
• malformed conversations
• irrelevant samples
• answers that violate the behavior you actually want
• train/test leakage
• automatically generated examples that were never checked

Fine-tuning amplifies your data decisions.

A clean thousand-example dataset can be more useful than a large dataset assembled without editorial control.

Step 5: create the QLoRA training script

Create:

train_lora.py

Add:

import torch
from datasets import load_dataset
from peft import LoraConfig
from transformers import BitsAndBytesConfig
from trl import SFTConfig, SFTTrainer

MODEL_ID = "meta-llama/Llama-3.1-8B-Instruct"

# Load conversational JSONL data.
dataset = load_dataset(
   "json",
   data_files="train.jsonl",
   split="train",
)

# Load the frozen base model in 4-bit.
quantization_config = BitsAndBytesConfig(
   load_in_4bit=True,
   bnb_4bit_quant_type="nf4",
   bnb_4bit_compute_dtype=torch.bfloat16,
   bnb_4bit_use_double_quant=True,
)

# Train LoRA adapters across the model's linear layers.
lora_config = LoraConfig(
   r=16,
   lora_alpha=32,
   lora_dropout=0.05,
   bias="none",
   task_type="CAUSAL_LM",
   target_modules="all-linear",
)

training_config = SFTConfig(
   output_dir="llama-3.1-8b-qlora",
   learning_rate=2e-4,
   num_train_epochs=1,
   per_device_train_batch_size=1,
   gradient_accumulation_steps=16,
   gradient_checkpointing=True,
   bf16=True,
   max_length=2048,
   packing=True,
   logging_steps=10,
   save_strategy="epoch",
   report_to="none",
)

trainer = SFTTrainer(
   model=MODEL_ID,
   args=training_config,
   train_dataset=dataset,
   quantization_config=quantization_config,
   peft_config=lora_config,
)

trainer.train()

trainer.save_model("llama-3.1-8b-lora-adapter")

This is a starting configuration, not a universal set of optimal hyperparameters.

Hugging Face currently recommends NF4 for 4-bit QLoRA-style training and supports applying LoRA to all-linear modules. Its TRL integration also notes that adapter training usually uses a higher learning rate than full-model fine-tuning.

What the QLoRA configuration is doing

The most important part is:

BitsAndBytesConfig(
   load_in_4bit=True,
   bnb_4bit_quant_type="nf4",
   bnb_4bit_compute_dtype=torch.bfloat16,
   bnb_4bit_use_double_quant=True,
)

This does not turn your finished adapter into a crude 4-bit approximation.

The base model is loaded in 4-bit and remains frozen. Computation uses BF16 where configured, while the LoRA parameters are trained separately.

That is the basic QLoRA pattern.

Hugging Face currently recommends NF4 for this type of training and supports nested, or double, quantization for further memory savings.

What does LoRA rank mean?

Our configuration uses:

r=16

The rank controls the size of the low-rank matrices LoRA learns.

A higher rank gives the adapter more trainable capacity, but it also adds parameters and memory use.

That does not mean you should automatically choose the highest rank your GPU can hold.

Common values include:

8
16
32
64

The useful value depends on the task and dataset.

A narrow formatting adaptation may not benefit from a large rank. A more complicated behavioral shift may.

Start with a modest value, evaluate it, and increase it because the results justify doing so rather than because larger numbers look safer.

Why target all linear layers?

The script uses:

target_modules="all-linear"

That follows the QLoRA-style approach supported by PEFT.

You can instead restrict LoRA to selected attention projections such as:

["q_proj", "v_proj"]

That reduces the number of trainable parameters further.

The tradeoff is straightforward: targeting more layers gives the adapter more places to learn changes, while targeting fewer layers reduces training cost.

PEFT's current documentation recommends all-linear for QLoRA-style training because it applies the adapter across the transformer's linear layers without requiring architecture-specific names.

Step 6: run the fine-tune

Start training:

python train_lora.py

In another terminal, monitor the GPU:

watch -n 1 nvidia-smi

Watch the training logs too.

You are looking for more than proof that the GPU is busy.

Check:

• whether loss decreases
• whether the run remains numerically stable
• whether memory use leaves some headroom
• whether checkpoints are written correctly
• whether training examples are being interpreted as expected

An error-free training run can still produce a bad model.

Do not optimize GPU utilization before checking the output

It is tempting to increase batch size until every megabyte of VRAM is occupied.

That is not the first objective.

Start with:

per_device_train_batch_size = 1
gradient_accumulation_steps = 16

Gradient accumulation lets the optimizer see an effectively larger batch without keeping every sample in GPU memory at the same time.

Once the workflow works, you can test a larger per-device batch and measure whether it improves training throughput.

The highest utilization percentage is not automatically the best configuration.

Sequence length has a large effect on memory

This tutorial sets:

max_length=2048

Llama 3.1 supports much longer contexts, but training at the model's maximum context is a completely different memory problem.

If most of your examples are a few hundred tokens long, training at 128K would waste extraordinary amounts of compute and memory.

Choose a sequence length that reflects your data.

If examples are being truncated, raise it carefully.

If almost every sample is short, a huge maximum length does not make the model better.

Packing can make short examples cheaper to train

We enable:

packing=True

Packing combines multiple short training examples into fuller sequences instead of padding every example separately to the maximum sequence length.

That can improve training efficiency when your dataset contains lots of short samples. TRL currently supports packing directly through SFTConfig.

If your examples already fill most of the sequence length, the benefit is smaller.

Step 7: save the adapter

At the end of training:

trainer.save_model("llama-3.1-8b-lora-adapter")

The saved directory contains the LoRA adaptation rather than another complete copy of Llama 3.1.

Keep the relationship clear:

Base model
meta-llama/Llama-3.1-8B-Instruct

+

Adapter
llama-3.1-8b-lora-adapter

Together, they represent your fine-tuned model.

This separation is one of LoRA's practical strengths. You can maintain several adapters against the same base model rather than copying the entire base checkpoint for every experiment.

Step 8: test the adapter

Create:

test_adapter.py

Add:

import torch
from peft import PeftModel
from transformers import (
   AutoModelForCausalLM,
   AutoTokenizer,
   BitsAndBytesConfig,
)

MODEL_ID = "meta-llama/Llama-3.1-8B-Instruct"
ADAPTER_PATH = "llama-3.1-8b-lora-adapter"

quantization_config = BitsAndBytesConfig(
   load_in_4bit=True,
   bnb_4bit_quant_type="nf4",
   bnb_4bit_compute_dtype=torch.bfloat16,
   bnb_4bit_use_double_quant=True,
)

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)

base_model = AutoModelForCausalLM.from_pretrained(
   MODEL_ID,
   quantization_config=quantization_config,
   device_map="auto",
)

model = PeftModel.from_pretrained(
   base_model,
   ADAPTER_PATH,
)

messages = [
   {
       "role": "user",
       "content": (
           "Rewrite this incident report as a concise technical summary: "
           "Users could not log in after a configuration change caused "
           "authentication requests to time out."
       ),
   }
]

inputs = tokenizer.apply_chat_template(
   messages,
   add_generation_prompt=True,
   return_tensors="pt",
).to(model.device)

with torch.inference_mode():
   outputs = model.generate(
       inputs,
       max_new_tokens=100,
       do_sample=False,
   )

response = tokenizer.decode(
   outputs[0][inputs.shape[-1]:],
   skip_special_tokens=True,
)

print(response)

Run:

python test_adapter.py

One output tells you almost nothing.

Test the adapter against a held-out evaluation set, including examples that were not in the training data.

Also compare those answers against the unmodified base model.

Without that baseline, you cannot tell whether fine-tuning helped.

Evaluation is part of fine-tuning

Training loss is useful.

It is not a product metric.

If your goal is structured extraction, measure extraction accuracy.

If your model should produce valid JSON, measure valid JSON.

If it should follow a particular support policy, construct a test set around that policy.

If you are adapting terminology, check terminology.

A vague judgment that responses "sound better" is weak evidence unless style is genuinely the task you are training.

Build the evaluation before you spend weeks tuning ranks, epochs, and learning rates.

More epochs are not automatically better

Our baseline uses:

num_train_epochs=1

That is intentionally conservative.

Small or repetitive datasets can overfit quickly. Training for five epochs because one epoch sounds insufficient can leave you with a model that reproduces training patterns extremely well and generalizes poorly.

Evaluate after the first run.

Then decide whether you need:

• more epochs
• more data
• better data
• a different rank
• a different learning rate
• a larger model

Hyperparameter tuning cannot rescue a bad dataset.

When should you use plain LoRA instead of QLoRA?

QLoRA is useful when GPU memory is the constraint.

Plain LoRA can make sense when the model fits comfortably at BF16 and you want to avoid quantizing the frozen base model during training.

On a 32GB RTX 5090, an 8B model gives you room to experiment with either approach.

For larger models, QLoRA becomes increasingly useful because the frozen model itself consumes much less VRAM.

Do not treat QLoRA as a universally better algorithm. It is a memory-efficient way of performing LoRA training.

Whether any difference matters to your application is an evaluation question.

LoRA does not make the base model yours

Training an adapter does not erase the license of the model underneath it.

Our example uses Llama 3.1, which is distributed under Meta's Llama 3.1 Community License. The model repository also sets conditions for redistribution and derivative models.

Before publishing or distributing an adapter, check:

• the base-model license
• the dataset licenses
• rights to the training material
• redistribution conditions
• naming and attribution requirements

"Open weights" is not a synonym for "no conditions."

How much does LoRA fine-tuning cost on a cloud GPU?

The useful calculation is:

GPU hourly rate × training runtime

Hivenet currently lists the RTX 5090 at €0.75 per GPU-hour with per-second billing.

For one RTX 5090:

GPU runtimeCompute cost30 minutes€0.381 hour€0.752 hours€1.504 hours€3.008 hours€6.00

Those are cost examples, not estimates of how long your fine-tune will take.

Training time changes with dataset size, sequence length, batch configuration, model size, number of epochs, and software stack.

Measure one representative run before forecasting a large training budget.

And stop the instance when the job is done.

LoRA is useful because experiments can stay small

The attraction of LoRA is sometimes described as cheap fine-tuning.

That undersells the useful part.

LoRA makes experimentation less destructive.

You can keep a base model stable, train one adapter for one task, another for a different task, compare them, discard poor runs, and retrain without producing a complete model checkpoint every time.

That changes how teams can work.

You do not have to decide that one fine-tune will become the model.

You can treat adaptation as an experiment with a defined dataset, configuration, evaluation, and adapter.

That is a healthier way to approach fine-tuning than assuming every model problem requires a large training job.

What to do after training

Once the adapter passes evaluation, you have several choices.

You can keep the adapter separate from the base model, merge it where the tooling and deployment path support that, or serve a LoRA-aware setup.

If you plan to expose the model through an inference server, see our vLLM guide.

For repeatable training jobs, you can also save the working environment as a reusable Compute setup rather than rebuilding dependencies for every experiment.

And if the model grows beyond a comfortable single-GPU workload, move to a larger GPU configuration because the workload requires it, not because multi-GPU sounds inherently better.

LoRA fine-tuning FAQ

What is LoRA fine-tuning?

LoRA, or Low-Rank Adaptation, is a parameter-efficient fine-tuning method. It freezes the original model weights and trains small low-rank matrices added to selected layers.

What is the difference between LoRA and QLoRA?

LoRA trains adapters while leaving the base model frozen. QLoRA also quantizes the frozen base model, typically to 4-bit, which reduces GPU memory requirements further.

Does LoRA change all the model weights?

No. The original model weights remain frozen. Training changes the LoRA adapter parameters.

Can you fine-tune Llama with LoRA?

Yes. Hugging Face PEFT and TRL support LoRA and QLoRA training for causal language models such as Llama.

Can you fine-tune Llama 3.1 8B on one GPU?

Yes, with a suitable parameter-efficient configuration. QLoRA makes an 8B model practical on a single 32GB GPU by loading the frozen base model at 4-bit and training LoRA adapters separately.

How much VRAM does LoRA fine-tuning need?

There is no single requirement. Model size, precision, batch size, sequence length, LoRA rank, target layers, and training configuration all affect VRAM use.

What LoRA rank should I use?

Ranks such as 8, 16, 32, and 64 are common starting points. Higher rank adds trainable capacity and parameters. Evaluate several values if the task warrants it instead of assuming the largest rank is best.

Is LoRA better than full fine-tuning?

It depends on the task. LoRA is much more resource-efficient and is often enough for task or behavior adaptation. Full fine-tuning gives you control over all model parameters but requires substantially more compute and memory.

Should I use LoRA or RAG?

Use LoRA when you want to change model behavior. Use RAG when the main requirement is retrieving factual information from documents or data that may change. They can also be combined.

How much training data do I need for LoRA?

There is no useful universal number. The right amount depends on the task, model, variety of inputs, and quality of the examples. A smaller clean dataset can outperform a much larger noisy one.

Can I use several LoRA adapters with one base model?

Yes. Keeping adapters separate lets you maintain multiple adaptations against the same underlying model, subject to the capabilities of your inference stack.

Do I need multiple GPUs for LoRA fine-tuning?

Not necessarily. Smaller models and QLoRA configurations can work on one capable GPU. Multi-GPU becomes useful when model size, sequence length, batch requirements, or training speed justify it.

‍