LoRA (low-rank adaptation) is a fine-tuning method that adapts a pretrained model by training a small add-on update instead of changing the original weights. It matters in AI engineering because it makes adapting a large model far cheaper in memory, and it produces a small adapter file you can swap in or out.
For example, consider one 4096 × 4096 weight matrix inside a model. Fully fine-tuning it would update 16,777,216 parameters. A rank-16 LoRA update to the same matrix trains 131,072 parameters, about 0.78% as many.
LoRA keeps the pretrained weights frozen. It learns the update as the product of two small matrices, which can be merged into the base weights or kept as a separate adapter.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Why LoRA matters
Full fine-tuning updates every weight in a model, which takes a lot of memory and produces a full-size copy for every variant. LoRA trains only a small fraction of parameters, so you can adapt a large model with far fewer resources.
LoRA is useful for:
- Adapting large models cheaply: Train task-specific behavior without the memory cost of updating every weight.
- Serving many variants: Keep one base model and load a different small adapter for each task.
- Rolling back safely: Remove an adapter to return to the unchanged base model.
- Comparing versions: Evaluate the base model and the adapted model side by side as separate releases.
Saying “we used LoRA” is not a quality claim. Rank, scaling, target modules, and training data decide how good an adapter is.
How LoRA works
A pretrained model contains large weight matrices. Fine-tuning normally changes those matrices directly.
LoRA leaves each original matrix frozen. It adds a learned update that is constrained to a low-rank form, meaning it is built from two much smaller matrices multiplied together.
This description follows the Hugging Face PEFT library’s LoRA conceptual guide.
Formula:
W = W0 + (α / r) × B A
Where:
- W is the effective weight matrix the model uses after adaptation.
- W0 is the frozen pretrained matrix, with shape d × k.
- d and k are the number of rows and columns in W0.
- B is a trainable matrix with shape d × r.
- A is a trainable matrix with shape r × k.
- r is the rank, chosen to be much smaller than both d and k.
- α (alpha) is a scaling constant.
The product B A has the same shape as W0, so it can be added directly to it. Because r is small, B and A together hold far fewer numbers than W0.
Training and inference
During training, W0 stays frozen. Only A and B receive gradients and change.
The α / r term sets how strongly the update acts relative to its rank. That is why rank and alpha are reported together.
At inference, you have two options:
- Merge the update: Add B A into W0. This adds no extra latency.
- Keep a separate adapter: Load the adapter alongside the base model. One base model can then serve many task-specific adapters.
Choosing target modules
You also choose which matrices get adapters. Common choices are the attention query and value projections, and sometimes all linear layers.
This choice changes both the adapter’s capacity and its cost. Adapting more matrices gives the update more room to change behavior, and it trains more parameters.
For background on the models being adapted, see deep learning model.
How to calculate LoRA’s trainable parameters
For one adapted matrix, the parameter counts are:
full update parameters = d × k
LoRA update parameters = r × (d + k)
Where:
- d × k counts every entry in the original matrix.
- r × (d + k) counts the entries in B (d × r) plus the entries in A (r × k).
The following configuration is an illustrative PEFT-style setup, not a recommendation. It adapts the query and value projections at rank 16.
config = { # illustrative PEFT-style configuration, not a recommendation
"r": 16, # rank of A and B
"lora_alpha": 32, # scaling = alpha / r = 2
"target_modules": ["q_proj", "v_proj"],
"task_type": "CAUSAL_LM",
}
The next snippet calculates the parameter counts for one 4096 × 4096 projection under that configuration.
d, k = 4096, 4096 # shape of one projection matrix
r = 16 # LoRA rank
alpha = 32 # LoRA scaling constant
full_update = d * k
lora_update = r * (d + k)
print(f"full update: {full_update:,} parameters")
print(f"LoRA update: {lora_update:,} parameters")
print(f"fraction: {lora_update / full_update:.2%}")
print(f"scaling: alpha / r = {alpha / r:g}")
Output:
full update: 16,777,216 parameters
LoRA update: 131,072 parameters
fraction: 0.78%
scaling: alpha / r = 2
This is why adapters are small. The adapter can add task behavior cheaply, but its capacity is deliberately limited.
If a task needs behavior that a low-rank update to two projections cannot express, you need more rank, more target modules, or a different method. Only evaluation on your own tasks can tell you which.
How to evaluate a LoRA adapter
Treat an adapter as a new model release. Start by pinning everything that affects its behavior:
- The base model revision.
- The rank, alpha, and target modules.
- The training-data version.
- The decoding settings used at evaluation time.
Then run the base model and the adapted model on the same evaluation dataset. Compare results per slice, not only the headline average.
Check for regressions outside the target task
Include slices the fine-tune was not aimed at, such as general instruction following, refusal behavior, and output formats that other features depend on. Adapter drift usually appears outside the target task first.
Keep the adapter’s training data out of its own test set. That leak makes adapters look better than they are. For more on keeping the two apart, see training data vs. evaluation data.
General guidance on LLM evaluation and evaluation metrics applies to adapters unchanged. After release, keep base and adapter traces reviewable through AI observability so you can compare behavior on real traffic.
LoRA vs. full fine-tuning, PEFT, and QLoRA
These terms are often used loosely. Each describes something different.
Full fine-tuning: Updates every weight. It offers the most capacity, the highest memory and storage cost per variant, and the greatest risk of disturbing general behavior.
LoRA: Trades some capacity for a frozen base, small adapters, and easy rollback. An adapter kept separate can simply be unloaded. Merged weights can be rolled back by reloading the unchanged base model.
PEFT (parameter-efficient fine-tuning): The family name for methods that train only a small number of parameters. It covers LoRA, prompt tuning, and other adapter methods. PEFT is not a synonym for LoRA.
QLoRA: Loads the frozen base model in 4-bit precision and trains LoRA adapters against it. It is a memory-saving combination, described in the Hugging Face PEFT quantization developer guide. Calling a plain LoRA run “QLoRA” misstates what was done.
For a walkthrough of the original study, see Arize’s paper discussion of LoRA: Low-Rank Adaptation of Large Language Models. A follow-up variant is covered in generalized LoRA (GLoRA).
Limitations of LoRA
- Limited capacity: Tasks far from the base model’s behavior, or tasks that need many unrelated behaviors at once, may underfit at low rank.
- Training data dominates: Narrow data can produce an adapter that handles the target format while degrading everything else. Evaluating only the target task will not show this.
- Unreliable for adding knowledge: Teaching facts through a small update is fragile compared with retrieving them at runtime.
- Merging can interfere: Adapters trained separately can conflict when merged, in ways none of them showed alone.
To manage these risks, version your training data, evaluate broadly, and keep the base model ready to ship.
Frequently asked questions
Does LoRA change the base model permanently?
Only if you merge the update and save the combined weights. The usual workflow keeps the base frozen and stores the adapter as a separate file. The base stays unchanged, and you can remove or swap the adapter.
What LoRA rank should I use?
There is no universal value. Start small, and increase rank when the task underfits, checking alpha and target modules at the same time. Let held-out task results decide, not training loss, and report rank with any result you share.
Is LoRA the same as PEFT or QLoRA?
No. PEFT is the family of parameter-efficient methods, and LoRA is one member of it. QLoRA also quantizes the frozen base model, so it makes different memory and quality trade-offs.
How do I know a LoRA adapter did not regress other tasks?
Run the base and adapted models on the same held-out sets, covering both the target task and adjacent behaviors. Compare results per slice, and keep the comparison with the release record. A target-task gain alone is not evidence that nothing else broke.
Putting LoRA to work
With every adapter, record the base model revision, rank, alpha, target modules, training-data version, and decoding settings. Without those details, results cannot be compared or reproduced.
Before release, compare the adapter with its base model on target and off-target slices. After release, tracing tools such as Arize Phoenix can capture outputs from both versions so you can compare them on real traffic.
