LoRA Fine-Tuning Guide

Why LoRA

Full fine-tuning a 7B model requires 50+ GB of VRAM. LoRA reduces trainable parameters by 1000x through low-rank decomposition.

Core Principle

Add low-rank matrices A and B alongside pre-trained weights:

$W' = W + \alpha \cdot AB$

Where $A \in \mathbb{R}^{d \times r}$, $B \in \mathbb{R}^{r \times k}$, and $r \ll d$.

from peft import LoraConfig
config = LoraConfig(r=16, lora_alpha=32, target_modules=["q_proj", "v_proj"])

QLoRA adds 4-bit quantization on top, enabling LLaMA-65B fine-tuning on a single RTX 3090. Many GPT-derivative models use LoRA, see GPT Series Evolution.