LoRA stands for Low-Rank Adaptation. It is the most widely used fine-tuning method today: instead of touching a model's tens of billions of weights, it freezes all of them and trains a small set of new parameters alongside.
The difference is a matter of orders of magnitude. Fully fine-tuning a 70B model takes a cluster and hundreds of gigabytes of memory; with LoRA, a single consumer GPU and an afternoon are often enough. The output file is typically tens to hundreds of megabytes and can be attached to the base model like a plugin, then detached when done.
Grab It in One Sentence First
LoRA doesn't modify the original model. It trains a thin layer of "corrections" beside it and adds them in at use time.
The everyday analogy is a camera filter. You don't disassemble the body and swap the sensor to shoot warmer tones — you put a filter in front of the lens. The camera is untouched, the output changes, and a different scene just calls for a different filter. LoRA is that filter for large models: one shared base, a different adapter per task.
Why So Few Parameters Suffice
The LoRA paper rests on a key observation: although the weight update during fine-tuning has a large shape, its intrinsic dimensionality is low — the effective change concentrates in a low-dimensional subspace. If so, there's no need for an update matrix as large as the original weights; the product of two thin matrices approximates it well enough.
The r in the diagram is the rank, typically a small value like 8, 16, or 32. If the original weight matrix is 4096×4096 (about 16 million parameters), a rank-16 LoRA needs only 4096×16 + 16×4096 ≈ 131,000 parameters, under a thousandth as many. Only A and B receive updates during training; the original weights don't even need gradients, so memory use drops immediately.
At inference there are two options: merge A×B back into the original weights for a model indistinguishable from an ordinary one with zero added latency, or keep them separate and combine at runtime — which lets a single base serve many adapters at once.
QLoRA: Fine-Tuning Large Models on Consumer GPUs
LoRA solved "how many parameters to train," but the base model still had to fit in memory. QLoRA closed that gap: quantize the base to 4 bits and freeze it, then train LoRA on top. Together, a fine-tuning job that used to require a multi-GPU cluster fits on one high-memory consumer card.
This is effectively the default configuration for open-source fine-tuning today, and the most common implementation behind "one-click fine-tuning" on hosted platforms.
What It's Good At — and Not
LoRA excels at adjusting behavior and style: locking a model into an output format, adopting an industry's terminology, imitating a particular voice, or sharpening a classification task. What these share is that the model already has the underlying capability; you're only fixing its tendency.
It is poor at injecting large bodies of new knowledge. If you want a model to know the contents of tens of thousands of internal documents, LoRA is the wrong tool and RAG is the right one: knowledge changes, retraining an adapter is far more work than updating an index, and hallucination doesn't disappear because you fine-tuned. A practical rule: *fine-tune to change how it says things, retrieve to supply what it says.*
On the image side, LoRA works almost identically and is even better known — the community's character LoRAs and style LoRAs are all small files attached to a shared diffusion model base.
Versus Other Approaches
Versus full fine-tuning. Full fine-tuning has more room to move and a higher ceiling when you have a lot of data and want to change behavior fundamentally, but it's expensive, prone to catastrophic forgetting, and requires storing a complete model per task. In most real scenarios LoRA gets close to full fine-tuning at one to two orders of magnitude less cost.
Versus prompt engineering. If a clearer prompt solves it, don't fine-tune — prompts change in seconds, fine-tuning takes hours. Fine-tuning pays off only when the prompt is already long and results are still unstable, or when you want to stop paying for that long prompt.
Versus distillation. Distillation trains a smaller model to approximate a larger one's behavior, aiming to cut inference cost. LoRA doesn't change model size; it changes behavior.
Where People Get It Wrong
"LoRA makes the model stronger." It makes the model fit your target task better, usually at a slight cost to general capability. The narrower the training data and the harder you train, the more pronounced that drift.
"Higher rank is better." Too high a rank approaches full fine-tuning's cost and overfits more easily. Common practice is to start at 8 or 16 and raise it only if validation results justify it, watching the learning rate and alpha ratio at the same time.
"More data is better." LoRA is extremely sensitive to data quality. A few hundred highly consistent, uniformly formatted examples usually beat tens of thousands of ragged ones. Most failures in practice are data problems, not hyperparameter problems.
"After fine-tuning, evaluation is done." The opposite. Test both the target task and prior capabilities afterwards to confirm you haven't sacrificed everything else for one metric — basic evals hygiene.
When It's Worth Doing
Rule things out in order: can a better prompt solve it? Can few-shot examples? Can RAG? Only when all three fail, and you have several hundred or more high-quality, consistently formatted examples, does LoRA become the answer.
One frequently overlooked motive is cost: train a two-thousand-token system prompt into an adapter and you stop paying for those tokens on every call. At sufficient volume that arithmetic can matter more than the quality gain, and it complements prompt caching as an option.