Freeze the Model, Train the Adapter

Full fine-tuning means updating every weight in a model — often billions of numbers — and storing gradients and optimizer state for every one of them. That's why "just fine-tune it" used to mean "rent a rack of GPUs." LoRA breaks that assumption: freeze the original weights entirely, and train a much smaller set of new parameters that sit alongside them.

Why freezing the weights is the actual trick

The core insight behind LoRA is that the update to a weight matrix during fine-tuning doesn't need to be full-rank. Instead of learning a full-size change matrix, LoRA represents that change as the product of two much smaller matrices — think of it as compressing the update itself, not the model. Multiply the two small matrices together and you get something the same shape as the original weight update, but with far fewer trainable parameters behind it.

Because the base weights are frozen, you don't need to store gradients or optimizer state for the entire model — only for the adapter. That's where the memory savings come from. It isn't a trick that shrinks the model on disk; it shrinks the amount of training state you have to keep in memory at once.

At inference time, the adapter's two matrices are multiplied together and added back into the original weights, so there's no separate "adapter layer" slowing down every forward pass. You get one merged set of weights that behaves like a fully fine-tuned model, produced by training a fraction of the parameters.

QLoRA: the same trick, plus quantization

QLoRA adds one more layer: it quantizes the frozen base model down to 4-bit precision before training the adapter on top of it. That shrinks the memory footprint of the base model itself — the part you're not even training — by roughly 4x compared to keeping it in 16-bit. QLoRA includes techniques to recover the precision lost from quantization, so accuracy stays close to standard LoRA while memory usage drops even further.

The upshot: LoRA already lets you fine-tune on modest hardware; QLoRA pushes that down to a single consumer GPU with a modest amount of VRAM, because the largest chunk of memory — the frozen base weights — is stored 4x smaller.

How to actually do it

1. Pick a base model and freeze it. Load it as-is; no weights are updated directly.
2. Attach LoRA adapters to the layers you want to adapt (commonly the attention projection layers). This adds a small number of new, trainable parameters.
3. (Optional, for QLoRA) Quantize the base model to 4-bit before attaching the adapter, so the frozen weights themselves take less memory.
4. Train only the adapter. The optimizer only tracks state for the adapter's parameters, which is why the memory savings are so large.
5. Merge the adapter back into the base weights for inference, or keep it separate if you want to swap adapters for different tasks.

The rank isn't the lever you think it is

It's tempting to assume a higher "rank" (the size of the adapter's small matrices) is what determines fine-tune quality, and that you should crank it up for better results. The evidence points the other way: which layers you apply the adapter to — ideally all of them, not just a subset — matters far more than the rank you pick. Performance stays roughly flat across a wide range of ranks (from single digits into the low hundreds). If your fine-tune is underperforming, check layer coverage before you chase a bigger rank number.

Power tricks

• Applying the adapter to only a few layers is the biggest quality lever — and it's easy to skip if you copy a default config that only targets attention layers on a subset of blocks.
• Don't assume a bigger rank fixes quality problems. It rarely does. Rank in the 8–256 range performs similarly; the difference you're chasing is more likely a layer-coverage or data-quality issue.
• Merge the adapter for deployment. If your serving stack doesn't support adapters natively, merge the LoRA matrices into the base weights before shipping, or you'll pay adapter-multiplication cost on every inference call.

Resources

• LoRA: Low-Rank Adaptation of Large Language Models (original paper)
• QLoRA: Efficient Finetuning of Quantized LLMs (original paper)
• Hugging Face PEFT — LoRA conceptual guide