Course: Course 3 — LLM Fine-Tuning Masterclass Module: FT09 — DoRA, rsLoRA, and Modern PEFT Duration: 60 minutes Level: Senior Engineer and above Prerequisites: FT08 (LoRA & QLoRA)
After completing this module, you will be able to:
α/r to α/√r, why this matters only at high rank (r ≥ 64), and why rsLoRA stacks with DoRA rather than competing with it.use_dora=True, use_rslora=True) and position QDoRA (quantized DoRA) as the leading quality/cost point.LoRA couples two things that full fine-tuning updates independently. DoRA separates them.
Recall the FT08 setup. LoRA approximates the weight update as ΔW = B·A, where A is r × d and B is d × r, with r typically 8–64. The frozen weight W₀ becomes W₀ + (α/r)·B·A. The optimizer learns A and B; the base W₀ is frozen.
This works — it is the democratizing technique of the field. But the DoRA authors (Liu et al., NVIDIA, arXiv:2402.09353, ICML 2024 Oral) asked a precise question: how does a LoRA update differ, structurally, from a full fine-tuning update? They performed a weight-decomposition analysis on every weight matrix, decomposing it into a magnitude component (the norm of each column / filter) and a direction component (the unit-normalized matrix). Then they tracked how each component moved during training, for full FT, for LoRA, and for a hypothetical ideal.
The finding: full fine-tuning updates magnitude and direction independently. Sometimes a filter's direction rotates a lot while its magnitude barely changes; sometimes the magnitude scales up while the direction holds. The two are decoupled. LoRA, by contrast, updates them proportionally — when the low-rank product B·A shifts a column, both its magnitude and its direction move together, locked in a fixed ratio determined by the rank and the scaling factor. LoRA cannot rotate a direction without also rescaling its magnitude, and vice versa.
This is the structural deficit DoRA names. It is roughly half of what full FT does, and it is exactly the half LoRA cannot express at any rank.
DoRA (Weight-Decomposed Low-Rank Adaptation) rewrites the update as:
W = m ⊙ (W₀ / ‖W₀‖) + ΔW_direction
In words: take the frozen base weight W₀, normalize it column-wise to get the pure direction, scale it back up by a learned per-column magnitude vector m, and apply a LoRA-style low-rank update only to the direction. The magnitude is handled by a separate, tiny, trainable vector m (one scalar per column). The direction is where the LoRA adapter lives.
The result is that the optimizer can now do the two things full FT does and LoRA cannot: change the direction of a filter while holding its magnitude fixed, and change the magnitude of a filter without rotating its direction. The expressive gap closes by roughly half — the half that was structurally inaccessible to vanilla LoRA. And it costs almost nothing: the magnitude vector is tiny (d parameters per adapted matrix), the LoRA adapter is the same size, and the forward/backward overhead is on the order of 5–10% more VRAM. After training, the magnitude and direction are folded back into the merged weight, so there is zero inference overhead — a merged DoRA adapter is identical in shape and speed to a merged LoRA.
Two regimes where DoRA's advantage is largest:
For a vanilla instruction-tuning SFT on a model already close to the target distribution, the DoRA advantage is real but smaller. For everything else — low rank, domain shift, QLoRA on a consumer card — DoRA is the new default.
A scaling-factor fix that unlocks high ranks. Complementary to DoRA, not competing.
There is a counterintuitive fact about LoRA: turning the rank up does not always help. You would expect a rank-256 adapter to dominate a rank-16 adapter — more parameters, more capacity. In practice, vanilla LoRA often gets worse or stalls at high rank. The Kalajdzievski rsLoRA paper (arXiv:2312.03732) named and fixed this.
The cause is the scaling factor. Vanilla LoRA scales the update by α / r. At rank 8 with α=16, the scale is 16/8 = 2.0. At rank 64 with the same α, the scale collapses to 16/64 = 0.25. As rank grows, the effective update shrinks toward zero, and the forward-pass activations of the adapter become vanishingly small relative to the base weight — the adapter is numerically present but functionally inert. Training becomes unstable: the optimizer is trying to learn a meaningful update through a 0.25× attenuator, and the gradients that flow back are correspondingly tiny. Higher rank buys capacity in theory and takes it away in practice.
rsLoRA changes one character: the scaling factor becomes α / √r. At rank 64 with α=16, the scale is now 16/√64 = 16/8 = 2.0 — the same as rank 8 under vanilla scaling. The forward pass stays numerically stable as rank climbs. This is why rsLoRA's full name is rank-stabilized LoRA: it stabilizes the forward-pass magnitudes across the rank spectrum, so high-rank adapters actually train the way their parameter count suggests they should.
The empirical payoff: at low rank (r ≤ 16) rsLoRA barely differs from vanilla LoRA — the scaling discrepancy is small. At high rank (r ≥ 64) rsLoRA is essential; vanilla LoRA wastes the capacity. The rule of thumb from the rsLoRA literature: if you are going above r=32, use rsLoRA. Below that, it does no harm but offers little.
This is the key combinatorial fact: rsLoRA and DoRA operate on different parts of the update. DoRA adds a magnitude/direction decomposition; rsLoRA fixes the scaling inside the direction's LoRA adapter. They are orthogonal. You use both: a DoRA decomposition whose direction-side adapter is scaled by α/√r instead of α/r. Empirically the combination beats either alone, especially at high rank. They are not competing methods you choose between — they are complementary knobs you turn together.
A drop-in alternative initialization that converges faster. Same parameter count as LoRA.
PiSSA (Principal Singular values and Singular Vectors Adaptation, arXiv:2404.02948, NeurIPS 2024) attacks a different part of the LoRA recipe: the initialization. Vanilla LoRA initializes A with random Gaussian noise and B with zeros, so the adapter starts as the identity (no update) and the optimizer builds the update from scratch.
PiSSA asks: what if we initialized the adapter from the most important directions already present in the weight matrix? It runs a fast SVD on the frozen W₀, splits it into principal components (top singular values/vectors) and residual components, and initializes the adapter from the principal part while freezing the residual as the new base. Concretely: the LoRA B and A matrices start seeded with the top-r singular vectors of W₀, scaled by the singular values, instead of zero and random.
The payoff: faster convergence and stronger final quality on NLU benchmarks, at the same parameter count and per-step cost as LoRA. The initialization takes only seconds (fast SVD), so it is nearly free to switch from LoRA to PiSSA. The tradeoff is that PiSSA modifies the frozen base (it reshuffles which part is frozen and which is trained), so the "merge back into the original base" workflow is slightly different — you must merge into the residual base, not the original. This is a one-time accounting cost, not a quality cost.
PiSSA is a drop-in alternative initialization, not a replacement for DoRA or rsLoRA. You can in principle combine them. In practice PiSSA is most often chosen when convergence speed matters (short training budgets) and the task is NLU-heavy (classification, extraction, NLI) where the principal components carry the task signal.
Full-parameter quality at near-LoRA memory. The bridge between PEFT and full FT.
GaLore (Gradient Low-Rank Projection, arXiv:2403.03507, ICML 2024) sits in a different category from the rest. DoRA, rsLoRA, and PiSSA are all still adapter methods — they freeze the base and train a small set of extra parameters. GaLore does the opposite in one sense and the same in another.
The observation: the dominant memory cost during training is not the weights and not the gradients — it is the optimizer state. Adam keeps two floating-point buffers (first and second moment) per parameter, in fp32. For a 7B model that is ~56 GB of optimizer state alone — the thing that makes full FT impossible on a 24 GB consumer card. LoRA sidesteps this by training only the adapter parameters, so the optimizer state is tiny.
GaLore's move: train ALL the parameters (full FT), but project the gradient into a low-rank subspace before the optimizer sees it. The model's weights are full-rank and get fully updated, but the optimizer state — the expensive part — lives in the low-rank subspace. Periodically the subspace is recomputed (via SVD of the recent gradient history) and rotated. The result: full-parameter learning at a fraction of the optimizer memory. GaLore fit a 7B full-parameter training run on a single 24 GB RTX 4090 — impossible for either vanilla full FT (not enough optimizer memory) or LoRA (LoRA freezes the base).
The use case in one sentence: "I want full-FT quality on a memory-limited node." If you have concluded (per FT10) that you actually need full-parameter FT — because the task is a large domain shift that adapters cannot fully capture — but you cannot afford the optimizer memory of vanilla full FT, GaLore is the bridge. It is slower than LoRA per step (the SVD refresh has cost) but it trains the whole model, which no adapter method does.
GaLore is not in competition with DoRA for the SFT-format-steering use case. DoRA wins there. GaLore is what you reach for when the decision tree (FT10) says "full FT" and the GPU says "no."
Where each modern PEFT method actually belongs. Not all papers become defaults.
The PEFT literature produces a new method every few weeks. Most do not graduate to default-status. Place the also-rans honestly:
| Method | One-line | Verdict |
|---|---|---|
| DoRA | Magnitude/direction decomposition | 2026 default. Closes ~half the LoRA-vs-full-FT gap. |
| rsLoRA | α/√r scaling |
2026 default at r ≥ 64. Stacks with DoRA. |
| PiSSA | SVD-based initialization | Strong alternative init. Fast convergence, NLU-heavy. |
| GaLore | Gradient low-rank projection for full FT | The full-FT bridge. Use when you need full FT on a small GPU. |
| VeRA | Shared frozen random matrices + per-layer scaling vectors | Niche / research. ~10× fewer params than LoRA but quality is not consistently there; useful only when adapter storage (not training memory) is the bottleneck. |
| AdaLoRA | Adaptive rank allocation during training | Largely superseded by DoRA. The problem AdaLoRA tried to solve (where to put the rank budget) is better solved by DoRA's decomposition. |
| MiSS | Claims SOTA on benchmarks | Experimental. Not widely reproduced or adopted. Treat as research-awareness, not a production default. |
The axis: standard (DoRA, rsLoRA, PiSSA, GaLore) vs niche/research (VeRA, AdaLoRA, MiSS). The reason this matters is the PEFT anti-pattern below: chasing novelty. A new paper claiming a 2% win on a benchmark is not a reason to switch your production stack. Reproducibility, ecosystem support, and consistent gains across tasks are.
DoRA + rsLoRA at appropriate rank. QDoRA for the quality/cost frontier.
After all the methods, here is what you actually run in 2026.
On a real GPU (16–24 GB+, fp16/bf16): DoRA + rsLoRA at rank 32–64, α set to roughly √r (or per the recipe). This is the default. It closes about half the gap to full FT, costs 5–10% more VRAM than vanilla LoRA, and merges to zero inference overhead.
On a consumer card (8–12 GB, quantized): QDoRA — DoRA on top of a 4-bit quantized base, the DoRA analogue of QLoRA. QDoRA is currently the leading quality/cost point in PEFT: full-FT-adjacent quality from a setup that fits a 7B-class model on a single consumer card. The decomposition adds negligible memory on top of QLoRA, and the quality lift over QLoRA is the same DoRA lift over LoRA.
When you genuinely need full FT (per FT10): GaLore, not vanilla full FT. Same quality target, fraction of the optimizer memory.
The shape of the decision: start with DoRA + rsLoRA. If the GPU is too small, drop to QDoRA. If DoRA is insufficient (measured, on a held-out eval — not vibes), the next step up is GaLore full FT, not vanilla LoRA at higher rank.
The configuration is two flags. Here is the canonical LoraConfig for the 2026 sweet spot:
from peft import LoraConfig, get_peft_model
config = LoraConfig(
r=64, # high rank — safe under rsLoRA
lora_alpha=64, # with rsLoRA, effective scale = alpha / sqrt(r)
use_dora=True, # enable magnitude/direction decomposition
use_rslora=True, # use alpha/sqrt(r) instead of alpha/r
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM",
target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj"],
)
model = get_peft_model(base_model, config)
Two flags. use_dora=True and use_rslora=True. That is the entire upgrade from the FT08 vanilla LoRA config to the 2026 default. No new training loop, no new optimizer, no new export pipeline — the merged artifact is shape-identical to a LoRA artifact and serves identically.
The PEFT field publishes a new method monthly. Each claims a win on some benchmark. Switching your production stack to every new method is how you spend a quarter and ship nothing. The discipline: only adopt a method after you have measured it against your current default on your held-out eval, on your data, with a controlled A/B. DoRA earned its default status through consistent gains across many tasks and models, reproduced by independent teams. A single-paper claim is not that.
The most common silent failure mode of "I just turned the rank up to 128 and it didn't help." Vanilla LoRA attenuates the update by 1/r — at rank 128 with α=16 the scale is 0.125, and the adapter is numerically present but functionally inert. The fix is one flag: use_rslora=True. If you are going above r=32 and you are not using rsLoRA, you are almost certainly wasting your rank budget and blaming the data.
The most expensive anti-pattern. Full FT costs 10–20× the memory and compute of DoRA, requires a much bigger GPU (or GaLore), produces a non-swappable artifact (no hot-swapping adapters), and on the majority of steering tasks delivers quality within a couple of points of DoRA. The FT10 decision tree exists to prevent this. The default is DoRA + rsLoRA; full FT is the exception you justify with a measured eval gap, not the starting point.
| Term | Definition |
|---|---|
| DoRA | Weight-Decomposed Low-Rank Adaptation (NVIDIA, arXiv:2402.09353). Decomposes the weight into magnitude + direction; applies LoRA to direction only. Closes ~half the LoRA-vs-full-FT gap at ~5–10% more VRAM. |
| rsLoRA | Rank-Stabilized LoRA (arXiv:2312.03732). Changes the scaling factor from α/r to α/√r, stabilizing high-rank training (r ≥ 64). |
| PiSSA | Principal Singular values and Singular Vectors Adaptation (arXiv:2404.02948). Initializes the adapter from the SVD principal components of the base weight; faster convergence on NLU. |
| GaLore | Gradient Low-Rank Projection (arXiv:2403.03507). Projects the gradient into a low-rank subspace so the optimizer state is small; enables full-parameter FT at near-LoRA memory. |
| VeRA | Vector-based Random Matrix Adaptation (arXiv:2310.11454). Frozen shared random matrices + small per-layer scaling vectors; ~10× fewer params than LoRA. Niche. |
| AdaLoRA | Adaptive rank allocation during training. Largely superseded by DoRA. |
| QDoRA | DoRA on top of a 4-bit quantized base (the DoRA analogue of QLoRA). Leading quality/cost point in 2026 PEFT. |
| Magnitude/direction decomposition | The DoRA split: the weight's per-column norm (magnitude) is separated from its unit-normalized form (direction), and each is adapted independently. |
| Rank-stability | The property that the forward-pass magnitudes of the adapter do not collapse as rank grows — what rsLoRA provides and vanilla LoRA lacks at r ≥ 64. |
| Optimizer state | The dominant memory cost of full-parameter training (Adam's first/second moment buffers). What GaLore compresses. |
See 07-lab-spec.md. The "LoRA vs DoRA" lab: fine-tune the same base on the same data with vanilla LoRA and with DoRA + rsLoRA, evaluate both on a held-out set, observe the quality delta. Consumer-GPU friendly (QDoRA-capable). The point is to feel the DoRA advantage on your own data — the only basis on which to adopt a new PEFT method.
α/√r scaling fix for high-rank LoRA.use_rslora=True.# Module FT09 — DoRA, rsLoRA, and Modern PEFT
**Course**: Course 3 — LLM Fine-Tuning Masterclass
**Module**: FT09 — DoRA, rsLoRA, and Modern PEFT
**Duration**: 60 minutes
**Level**: Senior Engineer and above
**Prerequisites**: FT08 (LoRA & QLoRA)
---
## Learning Objectives
After completing this module, you will be able to:
1. Explain what DoRA fixes about LoRA — the coupling of magnitude and direction updates — and defend the claim that DoRA closes roughly half the gap between LoRA and full fine-tuning at only ~5–10% more VRAM.
2. State why rsLoRA changes the scaling factor from `α/r` to `α/√r`, why this matters only at high rank (r ≥ 64), and why rsLoRA *stacks* with DoRA rather than competing with it.
3. Distinguish the modern PEFT family on a standard-vs-niche axis: DoRA + rsLoRA as the 2026 practical sweet spot, PiSSA as a drop-in initialization alternative, GaLore as the bridge to full-parameter FT, and VeRA / AdaLoRA / MiSS as niche or experimental.
4. Configure DoRA + rsLoRA in Hugging Face PEFT (`use_dora=True`, `use_rslora=True`) and position QDoRA (quantized DoRA) as the leading quality/cost point.
5. Avoid the three PEFT anti-patterns: chasing novelty without measuring, ignoring rank-stability at high rank, and reaching for full FT where DoRA would suffice.
---
# 9.1 — The Problem DoRA Solves
*LoRA couples two things that full fine-tuning updates independently. DoRA separates them.*
## Why LoRA leaves quality on the table
Recall the FT08 setup. LoRA approximates the weight update as `ΔW = B·A`, where `A` is `r × d` and `B` is `d × r`, with `r` typically 8–64. The frozen weight `W₀` becomes `W₀ + (α/r)·B·A`. The optimizer learns `A` and `B`; the base `W₀` is frozen.
This works — it is the democratizing technique of the field. But the DoRA authors (Liu et al., NVIDIA, arXiv:2402.09353, ICML 2024 Oral) asked a precise question: *how does a LoRA update differ, structurally, from a full fine-tuning update?* They performed a weight-decomposition analysis on every weight matrix, decomposing it into a **magnitude** component (the norm of each column / filter) and a **direction** component (the unit-normalized matrix). Then they tracked how each component moved during training, for full FT, for LoRA, and for a hypothetical ideal.
The finding: **full fine-tuning updates magnitude and direction independently.** Sometimes a filter's direction rotates a lot while its magnitude barely changes; sometimes the magnitude scales up while the direction holds. The two are decoupled. **LoRA, by contrast, updates them proportionally** — when the low-rank product `B·A` shifts a column, both its magnitude and its direction move together, locked in a fixed ratio determined by the rank and the scaling factor. LoRA *cannot* rotate a direction without also rescaling its magnitude, and vice versa.
This is the structural deficit DoRA names. It is roughly half of what full FT does, and it is exactly the half LoRA cannot express at any rank.
## The DoRA fix: decompose, then adapt direction only
DoRA (Weight-Decomposed Low-Rank Adaptation) rewrites the update as:
```
W = m ⊙ (W₀ / ‖W₀‖) + ΔW_direction
```
In words: take the frozen base weight `W₀`, normalize it column-wise to get the pure direction, scale it back up by a learned per-column magnitude vector `m`, and apply a LoRA-style low-rank update *only to the direction*. The magnitude is handled by a separate, tiny, trainable vector `m` (one scalar per column). The direction is where the LoRA adapter lives.
The result is that the optimizer can now do the two things full FT does and LoRA cannot: change the direction of a filter while holding its magnitude fixed, and change the magnitude of a filter without rotating its direction. The expressive gap closes by roughly half — the half that was structurally inaccessible to vanilla LoRA. And it costs almost nothing: the magnitude vector is tiny (`d` parameters per adapted matrix), the LoRA adapter is the same size, and the forward/backward overhead is on the order of 5–10% more VRAM. After training, the magnitude and direction are folded back into the merged weight, so **there is zero inference overhead** — a merged DoRA adapter is identical in shape and speed to a merged LoRA.
### Where DoRA wins most
Two regimes where DoRA's advantage is largest:
1. **Low ranks (r ≤ 16).** At low rank vanilla LoRA is most constrained — the proportional coupling dominates. DoRA's decoupling buys the most expressive headroom exactly where cheap adapters need it most.
2. **Domain shifts.** When the target distribution is far from the pretraining distribution (a new language family, a new modality, a heavy style transfer), full FT updates magnitude and direction aggressively. DoRA can approximate that; LoRA cannot. Empirically DoRA's lead over LoRA grows with the size of the distribution shift.
For a vanilla instruction-tuning SFT on a model already close to the target distribution, the DoRA advantage is real but smaller. For everything else — low rank, domain shift, QLoRA on a consumer card — DoRA is the new default.
---
# 9.2 — rsLoRA: Fixing High-Rank Instability
*A scaling-factor fix that unlocks high ranks. Complementary to DoRA, not competing.*
## The high-rank paradox
There is a counterintuitive fact about LoRA: **turning the rank up does not always help.** You would expect a rank-256 adapter to dominate a rank-16 adapter — more parameters, more capacity. In practice, vanilla LoRA often gets *worse* or stalls at high rank. The Kalajdzievski rsLoRA paper (arXiv:2312.03732) named and fixed this.
The cause is the scaling factor. Vanilla LoRA scales the update by `α / r`. At rank 8 with α=16, the scale is `16/8 = 2.0`. At rank 64 with the same α, the scale collapses to `16/64 = 0.25`. As rank grows, the effective update shrinks toward zero, and the forward-pass activations of the adapter become vanishingly small relative to the base weight — the adapter is numerically present but functionally inert. Training becomes unstable: the optimizer is trying to learn a meaningful update through a 0.25× attenuator, and the gradients that flow back are correspondingly tiny. Higher rank buys capacity in theory and takes it away in practice.
## The rsLoRA fix
rsLoRA changes one character: the scaling factor becomes `α / √r`. At rank 64 with α=16, the scale is now `16/√64 = 16/8 = 2.0` — the same as rank 8 under vanilla scaling. The forward pass stays numerically stable as rank climbs. This is why rsLoRA's full name is *rank-stabilized* LoRA: it stabilizes the forward-pass magnitudes across the rank spectrum, so high-rank adapters actually train the way their parameter count suggests they should.
The empirical payoff: at low rank (r ≤ 16) rsLoRA barely differs from vanilla LoRA — the scaling discrepancy is small. At high rank (r ≥ 64) rsLoRA is essential; vanilla LoRA wastes the capacity. The rule of thumb from the rsLoRA literature: **if you are going above r=32, use rsLoRA.** Below that, it does no harm but offers little.
### rsLoRA stacks with DoRA
This is the key combinatorial fact: rsLoRA and DoRA operate on different parts of the update. DoRA adds a magnitude/direction decomposition; rsLoRA fixes the scaling inside the direction's LoRA adapter. They are orthogonal. You use both: a DoRA decomposition whose direction-side adapter is scaled by `α/√r` instead of `α/r`. Empirically the combination beats either alone, especially at high rank. They are not competing methods you choose between — they are complementary knobs you turn together.
---
# 9.3 — PiSSA: Smarter Initialization
*A drop-in alternative initialization that converges faster. Same parameter count as LoRA.*
PiSSA (Principal Singular values and Singular Vectors Adaptation, arXiv:2404.02948, NeurIPS 2024) attacks a different part of the LoRA recipe: the initialization. Vanilla LoRA initializes `A` with random Gaussian noise and `B` with zeros, so the adapter starts as the identity (no update) and the optimizer builds the update from scratch.
PiSSA asks: what if we initialized the adapter from *the most important directions already present in the weight matrix*? It runs a fast SVD on the frozen `W₀`, splits it into principal components (top singular values/vectors) and residual components, and initializes the adapter from the *principal* part while freezing the residual as the new base. Concretely: the LoRA `B` and `A` matrices start seeded with the top-r singular vectors of `W₀`, scaled by the singular values, instead of zero and random.
The payoff: **faster convergence and stronger final quality on NLU benchmarks**, at the same parameter count and per-step cost as LoRA. The initialization takes only seconds (fast SVD), so it is nearly free to switch from LoRA to PiSSA. The tradeoff is that PiSSA modifies the *frozen* base (it reshuffles which part is frozen and which is trained), so the "merge back into the original base" workflow is slightly different — you must merge into the residual base, not the original. This is a one-time accounting cost, not a quality cost.
PiSSA is a drop-in alternative initialization, not a replacement for DoRA or rsLoRA. You can in principle combine them. In practice PiSSA is most often chosen when convergence speed matters (short training budgets) and the task is NLU-heavy (classification, extraction, NLI) where the principal components carry the task signal.
---
# 9.4 — GaLore: The Bridge to Full Fine-Tuning
*Full-parameter quality at near-LoRA memory. The bridge between PEFT and full FT.*
GaLore (Gradient Low-Rank Projection, arXiv:2403.03507, ICML 2024) sits in a different category from the rest. DoRA, rsLoRA, and PiSSA are all still *adapter* methods — they freeze the base and train a small set of extra parameters. GaLore does the opposite in one sense and the same in another.
The observation: the dominant memory cost during training is **not the weights and not the gradients — it is the optimizer state.** Adam keeps two floating-point buffers (first and second moment) per parameter, in fp32. For a 7B model that is ~56 GB of optimizer state alone — the thing that makes full FT impossible on a 24 GB consumer card. LoRA sidesteps this by training only the adapter parameters, so the optimizer state is tiny.
GaLore's move: **train ALL the parameters (full FT), but project the gradient into a low-rank subspace before the optimizer sees it.** The model's weights are full-rank and get fully updated, but the optimizer state — the expensive part — lives in the low-rank subspace. Periodically the subspace is recomputed (via SVD of the recent gradient history) and rotated. The result: full-parameter learning at a fraction of the optimizer memory. GaLore fit a 7B full-parameter training run on a single 24 GB RTX 4090 — impossible for either vanilla full FT (not enough optimizer memory) or LoRA (LoRA freezes the base).
### When to reach for GaLore
The use case in one sentence: *"I want full-FT quality on a memory-limited node."* If you have concluded (per FT10) that you actually need full-parameter FT — because the task is a large domain shift that adapters cannot fully capture — but you cannot afford the optimizer memory of vanilla full FT, GaLore is the bridge. It is slower than LoRA per step (the SVD refresh has cost) but it trains the whole model, which no adapter method does.
GaLore is not in competition with DoRA for the SFT-format-steering use case. DoRA wins there. GaLore is what you reach for when the decision tree (FT10) says "full FT" and the GPU says "no."
---
# 9.5 — The Standard-vs-Niche Axis
*Where each modern PEFT method actually belongs. Not all papers become defaults.*
The PEFT literature produces a new method every few weeks. Most do not graduate to default-status. Place the also-rans honestly:
| Method | One-line | Verdict |
| --- | --- | --- |
| **DoRA** | Magnitude/direction decomposition | **2026 default.** Closes ~half the LoRA-vs-full-FT gap. |
| **rsLoRA** | `α/√r` scaling | **2026 default at r ≥ 64.** Stacks with DoRA. |
| **PiSSA** | SVD-based initialization | **Strong alternative init.** Fast convergence, NLU-heavy. |
| **GaLore** | Gradient low-rank projection for full FT | **The full-FT bridge.** Use when you need full FT on a small GPU. |
| **VeRA** | Shared frozen random matrices + per-layer scaling vectors | **Niche / research.** ~10× fewer params than LoRA but quality is not consistently there; useful only when adapter *storage* (not training memory) is the bottleneck. |
| **AdaLoRA** | Adaptive rank allocation during training | **Largely superseded by DoRA.** The problem AdaLoRA tried to solve (where to put the rank budget) is better solved by DoRA's decomposition. |
| **MiSS** | Claims SOTA on benchmarks | **Experimental.** Not widely reproduced or adopted. Treat as research-awareness, not a production default. |
The axis: **standard** (DoRA, rsLoRA, PiSSA, GaLore) vs **niche/research** (VeRA, AdaLoRA, MiSS). The reason this matters is the PEFT anti-pattern below: chasing novelty. A new paper claiming a 2% win on a benchmark is not a reason to switch your production stack. Reproducibility, ecosystem support, and consistent gains across tasks are.
---
# 9.6 — The 2026 Practical Sweet Spot
*DoRA + rsLoRA at appropriate rank. QDoRA for the quality/cost frontier.*
After all the methods, here is what you actually run in 2026.
**On a real GPU (16–24 GB+, fp16/bf16):** DoRA + rsLoRA at rank 32–64, α set to roughly `√r` (or per the recipe). This is the default. It closes about half the gap to full FT, costs 5–10% more VRAM than vanilla LoRA, and merges to zero inference overhead.
**On a consumer card (8–12 GB, quantized):** QDoRA — DoRA on top of a 4-bit quantized base, the DoRA analogue of QLoRA. QDoRA is currently the leading quality/cost point in PEFT: full-FT-adjacent quality from a setup that fits a 7B-class model on a single consumer card. The decomposition adds negligible memory on top of QLoRA, and the quality lift over QLoRA is the same DoRA lift over LoRA.
**When you genuinely need full FT (per FT10):** GaLore, not vanilla full FT. Same quality target, fraction of the optimizer memory.
The shape of the decision: start with DoRA + rsLoRA. If the GPU is too small, drop to QDoRA. If DoRA is insufficient (measured, on a held-out eval — not vibes), the next step up is GaLore full FT, not vanilla LoRA at higher rank.
## How to configure DoRA + rsLoRA in PEFT
The configuration is two flags. Here is the canonical `LoraConfig` for the 2026 sweet spot:
```python
from peft import LoraConfig, get_peft_model
config = LoraConfig(
r=64, # high rank — safe under rsLoRA
lora_alpha=64, # with rsLoRA, effective scale = alpha / sqrt(r)
use_dora=True, # enable magnitude/direction decomposition
use_rslora=True, # use alpha/sqrt(r) instead of alpha/r
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM",
target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj"],
)
model = get_peft_model(base_model, config)
```
Two flags. `use_dora=True` and `use_rslora=True`. That is the entire upgrade from the FT08 vanilla LoRA config to the 2026 default. No new training loop, no new optimizer, no new export pipeline — the merged artifact is shape-identical to a LoRA artifact and serves identically.
---
# 9.7 — The Three Anti-Patterns
### Chasing novelty without measuring
The PEFT field publishes a new method monthly. Each claims a win on some benchmark. Switching your production stack to every new method is how you spend a quarter and ship nothing. The discipline: only adopt a method after you have measured it against your current default on *your* held-out eval, on *your* data, with a controlled A/B. DoRA earned its default status through consistent gains across many tasks and models, reproduced by independent teams. A single-paper claim is not that.
### Ignoring the rank-stability issue at high rank
The most common silent failure mode of "I just turned the rank up to 128 and it didn't help." Vanilla LoRA attenuates the update by `1/r` — at rank 128 with α=16 the scale is 0.125, and the adapter is numerically present but functionally inert. The fix is one flag: `use_rslora=True`. If you are going above r=32 and you are not using rsLoRA, you are almost certainly wasting your rank budget and blaming the data.
### Full fine-tuning where DoRA would suffice
The most expensive anti-pattern. Full FT costs 10–20× the memory and compute of DoRA, requires a much bigger GPU (or GaLore), produces a non-swappable artifact (no hot-swapping adapters), and on the majority of steering tasks delivers quality within a couple of points of DoRA. The FT10 decision tree exists to prevent this. The default is DoRA + rsLoRA; full FT is the exception you justify with a measured eval gap, not the starting point.
---
## Key Terms
| Term | Definition |
| --- | --- |
| **DoRA** | Weight-Decomposed Low-Rank Adaptation (NVIDIA, arXiv:2402.09353). Decomposes the weight into magnitude + direction; applies LoRA to direction only. Closes ~half the LoRA-vs-full-FT gap at ~5–10% more VRAM. |
| **rsLoRA** | Rank-Stabilized LoRA (arXiv:2312.03732). Changes the scaling factor from `α/r` to `α/√r`, stabilizing high-rank training (r ≥ 64). |
| **PiSSA** | Principal Singular values and Singular Vectors Adaptation (arXiv:2404.02948). Initializes the adapter from the SVD principal components of the base weight; faster convergence on NLU. |
| **GaLore** | Gradient Low-Rank Projection (arXiv:2403.03507). Projects the gradient into a low-rank subspace so the optimizer state is small; enables full-parameter FT at near-LoRA memory. |
| **VeRA** | Vector-based Random Matrix Adaptation (arXiv:2310.11454). Frozen shared random matrices + small per-layer scaling vectors; ~10× fewer params than LoRA. Niche. |
| **AdaLoRA** | Adaptive rank allocation during training. Largely superseded by DoRA. |
| **QDoRA** | DoRA on top of a 4-bit quantized base (the DoRA analogue of QLoRA). Leading quality/cost point in 2026 PEFT. |
| **Magnitude/direction decomposition** | The DoRA split: the weight's per-column norm (magnitude) is separated from its unit-normalized form (direction), and each is adapted independently. |
| **Rank-stability** | The property that the forward-pass magnitudes of the adapter do not collapse as rank grows — what rsLoRA provides and vanilla LoRA lacks at r ≥ 64. |
| **Optimizer state** | The dominant memory cost of full-parameter training (Adam's first/second moment buffers). What GaLore compresses. |
---
## Lab Exercise
See `07-lab-spec.md`. The "LoRA vs DoRA" lab: fine-tune the same base on the same data with vanilla LoRA and with DoRA + rsLoRA, evaluate both on a held-out set, observe the quality delta. Consumer-GPU friendly (QDoRA-capable). The point is to *feel* the DoRA advantage on your own data — the only basis on which to adopt a new PEFT method.
---
## References
1. **Liu et al. (2024)** — *DoRA: Weight-Decomposed Low-Rank Adaptation of Large Language Models*. arXiv:2402.09353. ICML 2024 (Oral). The magnitude/direction decomposition; closes ~half the LoRA-vs-full-FT gap.
2. **Kalajdzievski (2023)** — *A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA* (rsLoRA). arXiv:2312.03732. The `α/√r` scaling fix for high-rank LoRA.
3. **Meng et al. (2024)** — *PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models*. arXiv:2404.02948. NeurIPS 2024. SVD-based initialization for faster convergence.
4. **Zhao et al. (2024)** — *GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection*. arXiv:2403.03507. ICML 2024. Full-parameter FT at near-LoRA memory.
5. **Kopiczko, Blankevoort, Asano (2023)** — *VeRA: Vector-based Random Matrix Adaptation*. arXiv:2310.11454. Shared frozen random matrices; extreme parameter reduction.
6. **NVIDIA Developer Blog** — *Introducing DoRA: A High-Performing Alternative to LoRA for Fine-Tuning*. The accessible write-up of the DoRA decomposition.
7. **Hugging Face PEFT** — *rsLoRA: Lightweight PEFT method* (blog by Damjan K.). The practical guide to `use_rslora=True`.
8. **FT08 — LoRA & QLoRA** — The prerequisite. FT09 builds directly on the LoRA math and the QLoRA memory trick introduced there.
9. **FT10 — Full FT vs PEFT: The Decision** — The next module. Where DoRA sits on the decision tree, and when you escalate from DoRA to GaLore full FT.