DoRA, rsLoRA, and Modern PEFT

Module FT09 · Course 3 — LLM Fine-Tuning Masterclass

60 minutes · the second module of Pillar 2 — what grew on the LoRA trunk in 2024, and which branches you use in 2026

DoRA + rsLoRA closes ~half the gap between vanilla LoRA and full FT — at ~5–10% more VRAM, and zero inference overhead after merge. Two flags. That is the new default.

Pillar 2 — Parameter-Efficient Fine-Tuning

The structural deficit DoRA names

DoRA's weight-decomposition analysis (NVIDIA, arXiv:2402.09353): full FT and LoRA update weights in structurally different geometry.

Method	Magnitude update	Direction update
Full FT	independent	independent — decoupled
LoRA	∝ coupled	∝ coupled — locked ratio
DoRA	vector m	LoRA on direction — decoupled

LoRA cannot rotate a direction without rescaling its magnitude, and vice versa. That is ~half of what full FT does — the half LoRA cannot express at any rank. Turning rank up does not fix it; the coupling is structural.

The DoRA fix: decompose, adapt direction only

Rewrite the update as magnitude × direction, with LoRA applied only to the direction:

W = m ⊙ (W₀ / ‖W₀‖) + ΔW_direction
        ↑              ↑              ↑
   per-column      unit-         LoRA adapter
   magnitude       normalized    (B·A), direction only
   vector m        frozen base

Quality: closes ~half the LoRA↔full-FT gap. Optimizer can now change direction w/o rescaling, and vice versa.

Cost: ~5–10% more VRAM (tiny magnitude vectors). After merge → zero inference overhead; shape-identical to LoRA.

Where DoRA wins most

DoRA's advantage is not uniform across tasks. Two regimes where it is largest:

1. Low rank (r ≤ 16). Vanilla LoRA is most constrained by the coupling here — the proportional lock dominates. DoRA's decoupling buys the most headroom exactly where cheap adapters need it. Huge for QLoRA-class work on consumer cards.

2. Domain shifts. When the target is far from pretraining (new language, new modality, heavy style transfer), full FT updates magnitude & direction aggressively. DoRA approximates that; LoRA cannot. The lead grows with the shift size.

Corollary: for vanilla instruction-tuning SFT close to the base distribution, the DoRA advantage is real but smaller. DoRA is a large win where you need it most, a small win where LoRA was already adequate. Read benchmarks with that nuance.

rsLoRA — fix high-rank instability

The paradox: turning rank UP in vanilla LoRA often makes it worse. Cause = the scaling factor.

Rank, α=16	Vanilla LoRA α/r	rsLoRA α/√r
r = 8	16/8 = 2.0	16/√8 = 5.66
r = 64	16/64 = 0.25 ← inert	16/√64 = 2.0
r = 128	16/128 = 0.125 ← inert	16/√128 = 1.41

At high rank vanilla α/r collapses the update toward zero — adapter is numerically present but functionally inert. Fix: use_rslora=True. Rule of thumb: above r=32, use rsLoRA.

DoRA + rsLoRA STACK — they are orthogonal

DoRA edits

The magnitude/direction split on top of the frozen base. Adds the decoupling full FT has.

rsLoRA edits

The scaling factor inside the direction-side LoRA adapter. α/r → α/√r.

Different parts of the update. Orthogonal knobs. The combination beats either alone — especially at high rank, where DoRA's lead shrinks but rsLoRA's grows. You use BOTH, you do not choose between them.

This is why the PEFT menu is not single-choice. DoRA + rsLoRA is the stack; PiSSA is an init layer you add on top.

PiSSA & GaLore — alternative init + the full-FT bridge

PiSSA (arXiv:2404.02948, NeurIPS '24)

Drop-in initialization alternative. Seeds adapter B,A from the principal singular vectors of W₀ (fast SVD) instead of random/zero. Faster convergence, strong on NLU. Same param count.

Tradeoff: reshuffles frozen vs trained → merge into the residual base, not the original.

GaLore (arXiv:2403.03507, MLSys '24)

NOT an adapter. Full-parameter FT with the gradient projected into a low-rank subspace — so the optimizer state (the ~56 GB Adam buffers for a 7B) is compressed.

7B full-param training on a single 24 GB RTX 4090. Use case: "I want full-FT quality on a memory-limited node."

GaLore is not competing with DoRA for SFT steering. DoRA wins there. GaLore is what you reach for when the FT10 decision says full FT and the GPU says no.

The standard-vs-niche axis

Most PEFT papers do not graduate to default. Place them honestly:

Method	One-line	Verdict
DoRA	magnitude/direction decomposition	2026 default
rsLoRA	α/√r scaling	default at r ≥ 64
PiSSA	SVD-based init	strong alt init
GaLore	gradient low-rank projection	the full-FT bridge
VeRA	shared random matrices, ~10× fewer params	niche — storage bottleneck only
AdaLoRA	adaptive rank allocation	superseded by DoRA
MiSS	claims SOTA	experimental, not reproduced

The axis is not recency — it is consistent gains + independent reproduction + ecosystem support. DoRA earned it. The others did not.

The 2026 sweet spot — two flags

Real GPU (16–24 GB, bf16)

DoRA + rsLoRA, r=32–64, α≈√r. The default.

Consumer card (<12 GB, 4-bit)

QDoRA — DoRA on a quantized base. Leading quality/cost.

Decision says full FT (FT10)

GaLore — not vanilla full FT.

The config

config = LoraConfig(
    r=64,
    lora_alpha=64,
    use_dora=True,      # ← the
    use_rslora=True,    # ← upgrade
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
    target_modules=["q_proj","k_proj",
      "v_proj","o_proj",
      "gate_proj","up_proj",
      "down_proj"],
)

Two flags — use_dora=True and use_rslora=True — are the entire upgrade from FT08's vanilla LoRA config. No new training loop, optimizer, or export pipeline.

The three anti-patterns

1. Chasing novelty without measuring. Switching the production stack to every new paper-claiming-a-win. Adopt only after a controlled A/B on your held-out eval, your data. DoRA earned default status through consistent gains reproduced by independent teams. A single-paper claim is not that.

2. Ignoring rank-stability at high rank. "I set r=128 and it didn't help, the data must be bad." At r=128 vanilla α/r = 0.125 — adapter is inert. Fix: one flag, use_rslora=True. Above r=32 without rsLoRA = wasting your rank budget.

3. Full FT where DoRA would suffice. 10–20× the memory/compute, non-swappable artifact, and on most steering tasks within a couple points of DoRA. FT10's decision tree exists to prevent this. Default is DoRA + rsLoRA; full FT is the justified exception, not the starting point.

The lab — LoRA vs DoRA, controlled A/B

Fine-tune the same base on the same data with vanilla LoRA and with DoRA + rsLoRA. Eval both on a held-out set. Observe the delta.

The discipline of a PEFT A/B: change exactly one thing (the adapter method), hold everything else fixed — base, data, rank, epochs, batch size, LR. Otherwise the delta is confounded and you cannot attribute it to DoRA.

Expect DoRA ahead — especially at low rank (r=16) where the decoupling buys the most headroom. The gap is typically a few ROUGE-L points, not a blowout. The direction is the finding. If your task is close to the base distribution, the gap may be small — itself a result.

Consumer-GPU friendly (QDoRA-capable). The point: feel the DoRA advantage on your own data — the only basis on which to adopt a new PEFT method.

What you can now do

Explain what DoRA fixes about LoRA — the coupling of magnitude and direction — and defend the "~half the gap at ~5–10% more VRAM" claim.
State why rsLoRA swaps α/r for α/√r, why it matters only at high rank, and why it stacks with DoRA rather than competing.
Distinguish the modern PEFT family on a standard-vs-niche axis: DoRA+rsLoRA (default), PiSSA (alt init), GaLore (full-FT bridge), VeRA/AdaLoRA/MiSS (niche).
Configure DoRA + rsLoRA in PEFT (use_dora=True, use_rslora=True) and position QDoRA as the leading quality/cost point.
Avoid the three anti-patterns: chasing novelty, ignoring rank-stability, full-FT-where-DoRA-suffices.

Next: FT10 — Full FT vs PEFT: The Decision

Where DoRA sits on the decision tree, and when you escalate from DoRA to GaLore full FT.