LoRA

Fine-tuning a large model the usual way updates all of its weights. With billions of parameters that needs a lot of GPU memory, and leaves you with a full copy of the model for every task. LoRA (Low-Rank Adaptation) avoids both: it freezes the original weights and trains only a small set of extra ones, an adapter, that is added on top.

The trick is in the shape. The weights of a layer form a big matrix, and fine-tuning changes it by an update matrix of the same size. LoRA bets that this update does not need all that freedom and writes it as the product of two thin matrices. For a 4096 x 4096 layer, instead of roughly 16.8 million numbers you train two strips of, say, 4096 x 8 each: about 65,000. That small width, 8 here, is the rank, the main setting you choose. When training ends, the update can be merged into the original weights, so the model runs as fast as before.

Researchers at Microsoft introduced LoRA in 2021. On GPT-3 they reported cutting the trainable parameters by about 10,000 times and the GPU memory needed by about 3 times, with quality close to full fine-tuning. QLoRA (2023) went further by keeping the frozen model quantized to 4 bits, which was enough to fine-tune a 65-billion-parameter model on a single 48 GB GPU.

LoRA is everywhere around open-weights models. Adapters are small files, megabytes rather than gigabytes, so you can keep one base model and swap adapters per task or per customer. In image generation, people share LoRAs that teach a diffusion model a style, a character or a product. The trade-off: a low-rank update can learn less than full fine-tuning, which shows on big changes, like heavy training on a whole new domain.